-- Logs begin at Fri 2022-10-28 10:22:52 UTC, end at Fri 2022-10-28 11:25:02 UTC. -- Oct 28 10:46:43.475454 dv-cinder-zed-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 28 10:46:43.484194 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Oct 28 10:46:43 2022] *** Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: nodename: dv-cinder-zed-3-iscsi Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: machine: x86_64 Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: clock source: unix Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: pcre jit disabled Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: detected number of CPU cores: 4 Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: current working directory: / Oct 28 10:46:43.486199 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: detected binary path: /usr/bin/uwsgi-core Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: your processes number limit is 47680 Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: your memory page size is 4096 bytes Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: detected max file descriptor number: 2048 Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: lock engine: pthread robust mutexes Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: thunder lock: enabled Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 28 10:46:43.486774 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 28 10:46:43.525957 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: Python main interpreter initialized at 0x560e188499c0 Oct 28 10:46:43.525957 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: python threads support enabled Oct 28 10:46:43.525957 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: your server socket listen backlog is limited to 100 connections Oct 28 10:46:43.525957 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: your mercy for graceful operations on workers is 90 seconds Oct 28 10:46:43.526349 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: mapped 403077 bytes (393 KB) for 2 cores Oct 28 10:46:43.526349 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: *** Operational MODE: preforking *** Oct 28 10:46:43.526588 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: *** uWSGI is running in multiple interpreter mode *** Oct 28 10:46:43.526588 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: spawned uWSGI master process (pid: 81292) Oct 28 10:46:43.526782 dv-cinder-zed-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 28 10:46:43.529437 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: spawned uWSGI worker 1 (pid: 81305, cores: 1) Oct 28 10:46:43.529437 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: spawned uWSGI worker 2 (pid: 81306, cores: 1) Oct 28 10:46:43.529437 dv-cinder-zed-3-iscsi devstack@keystone.service[81292]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 28 10:46:47.802429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 10:46:47.802429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: from cryptography.utils import int_from_bytes Oct 28 10:46:47.803321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 10:46:47.803321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: from cryptography.utils import int_from_bytes Oct 28 10:46:47.803443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 10:46:47.803443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: from cryptography.utils import int_from_bytes Oct 28 10:46:47.803443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 10:46:47.803443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: from cryptography.utils import int_from_bytes Oct 28 10:46:47.854440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 10:46:47.854632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 10:46:47.854815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] command line args: [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 10:46:47.855084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 10:46:47.855172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 10:46:47.855274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ================================================================================ {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 10:46:47.855352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 10:46:47.855478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] command line args: [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 10:46:47.855560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] admin_token = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.855749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 10:46:47.855825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] config_dir = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.855930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ================================================================================ {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 10:46:47.856017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] config_source = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] admin_token = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] debug = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] config_dir = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] config_source = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.856940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] debug = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.857913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_config_append = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_dir = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_config_append = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_file = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_dir = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.858964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_file = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.859988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.860920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.861866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.46/identity {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] publish_errors = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.46/identity {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.862962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] publish_errors = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.863940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] standard_threads = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] standard_threads = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] use_journal = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] use_json = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.864950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] use_stderr = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] use_journal = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] use_syslog = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] use_json = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] use_stderr = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] use_syslog = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.865813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 10:46:47.865957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.866932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.867865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.868998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.869954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.870926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.871977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.872838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.873819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.874938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.875996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.876931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.877936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.878911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.879870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.880944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.881965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.882959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.883984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.884923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.885857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.886852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.887963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.888960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.889898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.890894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.external = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.external = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.891828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.password = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] auth.token = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.password = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] auth.token = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.892967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.893910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.894968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.895935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.896898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.897929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.898118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.898205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.898361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.898437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.899945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.900899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.901861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.902981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.903941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.904955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.905933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.906917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.907897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.908961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.909910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.910934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.911917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.912956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.913909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.914091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.914169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.914863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.914947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.915989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.916977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.917907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.918901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.919995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.920972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.921956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.922997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.923999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.924872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.925963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] role.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] role.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] role.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] role.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.926970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.927944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.928926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.929893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.930917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.931972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.932954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.933959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.934980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.935951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.936905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.937875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.938932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.connection = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.connection = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.939937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.940938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.941947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.942957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.943967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.944090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.944220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81305) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 10:46:47.944346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 10:46:47.944346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81306) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 10:46:47.971628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.971858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.971997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.972123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.972264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.972385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.981282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:47.981498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.009237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.009471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.009597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.009818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.062976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.063177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.112344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.112522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.112651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.112775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.267976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.268868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.269005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.269098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.269220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.292558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.292814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.292908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.293908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.294009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.294156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.305125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.305464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.317779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.317919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.318876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.319014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.319097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81305) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.319229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81306) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 10:46:48.336755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.336970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 10:46:48.337176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.337323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.337404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 10:46:48.337529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 10:46:48.337720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.337799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.337924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 10:46:48.338084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.338223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.338353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.338487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.338617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.338750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.339009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.348306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.348565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.348685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.348846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.348941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.349121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.349205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.349409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.356411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.356678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.359351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.359606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.359689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.359893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.359974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.360814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.361893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.362110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.362193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.362386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.385254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.385479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.386742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.386972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.387054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.387255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.391710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.391919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.392000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.392210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.392302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.392496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.399278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.399469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.399552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.399755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.405179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.405386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.405469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.405687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.405768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.406937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.407128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.407208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.407401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.422702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.422921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.423004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.423200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.426861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.427073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.430648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.430840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.430929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.431137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.435175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.435375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.438356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.438614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.441596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.441834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.441918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.442817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.443027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.443108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.443321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.459418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.459609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.459693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.459901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.459982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.460173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.460255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.460440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.467107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.467400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.468663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.468936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.470223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.470498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.470581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.470777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.473147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.473424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.473506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.473710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.473792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.474008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.474093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.474306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.474405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.474692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.485809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.486106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.486191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.486398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.486481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.486681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.486765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.486960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.487927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.503529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.503912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.506457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.506857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.509896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.510321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.511556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.511790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.513047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.513256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.513338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.513546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.513724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.513942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.520115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.520275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.521576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.521721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.524520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.524673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.524916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.525079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.525208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.525340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.525492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.525621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.533298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.533386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.533594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.533678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.533891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.533993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.540829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81306) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.540925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81305) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 10:46:48.541115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.541946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.542888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.543987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.544204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.544306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.544587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81306) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.544668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.544894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81305) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 10:46:48.579427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560e188499c0 pid: 81306 (default app) Oct 28 10:46:48.579849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560e188499c0 pid: 81305 (default app) Oct 28 10:46:48.582686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e36a3e-d28a-448f-86ca-dd5b8b554a48 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:48.582975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e36a3e-d28a-448f-86ca-dd5b8b554a48 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:48.583251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e36a3e-d28a-448f-86ca-dd5b8b554a48 None None] PATH_INFO: `/v3` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:48.587493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1/1] 10.222.0.46 () {50 vars in 791 bytes} [Fri Oct 28 10:46:48 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:46:55.296319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:55.296799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:55.296902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:55.301082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1/2] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:46:55 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:46:55.307055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:55.307479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:55.307617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:55.377548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: INFO dbcounter [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] Registered counter for database keystone Oct 28 10:46:55.396639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG oslo_db.sqlalchemy.engines [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81306) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 28 10:46:55.402886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writer thread running {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 10:46:55.579928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG passlib.handlers.bcrypt [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81306) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 10:46:55.580324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG passlib.handlers.bcrypt [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81306) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 10:46:55.637908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:46:55.639517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:46:55.764661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 2/3] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:46:55 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 459 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 28 10:46:56.210674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:46:56.285113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: INFO dbcounter [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] Registered counter for database keystone Oct 28 10:46:56.307578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG oslo_db.sqlalchemy.engines [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81305) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 28 10:46:56.313741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writer thread running {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 10:46:56.437918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76e2bc0-9b4b-481f-8d45-4ee293b01ccc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:46:56.442255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:46:56.443258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:56.443760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:56.444062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:56.452945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:46:56.504479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING keystone.common.rbac_enforcer.enforcer [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 10:46:56.504767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b55c64e-7f29-410f-8920-fe0b734c8cdd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:46:56.514709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 2/4] 10.222.0.46 () {58 vars in 1324 bytes} [Fri Oct 28 10:46:56 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 10:46:56.520728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:46:56.538645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cf257-69d4-4e7c-89b8-9b49a01554c4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:46:56.543003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:46:56.544100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:56.544386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:56.544661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] PATH_INFO: `/v3/projects/2b351b41bbb64be2822e6354d987b945` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:56.546272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] RBAC: Authorizing `identity:get_project(project_id=2b351b41bbb64be2822e6354d987b945)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:46:56.600063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING keystone.common.rbac_enforcer.enforcer [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 10:46:56.600415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08151270-1bac-41fa-9e19-ff9ff0b75c91 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:46:56.605164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 3/5] 10.222.0.46 () {56 vars in 1207 bytes} [Fri Oct 28 10:46:56 2022] GET /identity/v3/projects/2b351b41bbb64be2822e6354d987b945 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:46:58.712415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6a23a-e583-4aaf-9a2a-3d0204ef1497 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:58.712981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6a23a-e583-4aaf-9a2a-3d0204ef1497 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:58.712981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b6a23a-e583-4aaf-9a2a-3d0204ef1497 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:58.717102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 3/6] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:46:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:46:58.722524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:58.722903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:58.723074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:58.744015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:46:58.745519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:46:58.753206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 4/7] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:46:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 28 10:46:59.025328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b6a23a-e583-4aaf-9a2a-3d0204ef1497 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:46:59.037189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b6a23a-e583-4aaf-9a2a-3d0204ef1497 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:46:59.041996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:46:59.042591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:59.042868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:59.043145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:59.050690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:46:59.052459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ca0f7-ff98-421a-afac-b29e5c4f2e49 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:46:59.061097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 4/8] 10.222.0.46 () {58 vars in 1324 bytes} [Fri Oct 28 10:46:59 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 10:46:59.066571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:46:59.075630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1441536-8d78-4598-b5d4-71b8b86c3a07 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:46:59.080087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:46:59.080741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:46:59.081026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:46:59.081303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:46:59.082657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:46:59.084484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc39428b-b8e0-4915-99d2-761f3e6f9c62 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:46:59.086587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 5/9] 10.222.0.46 () {56 vars in 1198 bytes} [Fri Oct 28 10:46:59 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:01.285890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b0a9f8-a6d0-4e7f-9e4f-78f8348ce850 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.286349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b0a9f8-a6d0-4e7f-9e4f-78f8348ce850 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.286506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b0a9f8-a6d0-4e7f-9e4f-78f8348ce850 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.287850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 5/10] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:01.293128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.293557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.293652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.296187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2809f52c-9a24-402a-b482-dfdbbdf98c8d None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.296474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2809f52c-9a24-402a-b482-dfdbbdf98c8d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.296755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2809f52c-9a24-402a-b482-dfdbbdf98c8d None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.297957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 6/11] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:01.303429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.303719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.304000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.312359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:01.313519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:01.346248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG passlib.handlers.bcrypt [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81305) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 10:47:01.346642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG passlib.handlers.bcrypt [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81305) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 10:47:01.393768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 6/12] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:01.396255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:01.397859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:01.496000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 7/13] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:01.660158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:01.685011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:01.760663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:01.764613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e26aaf-6b40-441b-bd1d-530667a72b8f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:01.799232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:01.799857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.800133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.800405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.812244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:01.821050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:01.822058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:01.822879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 7/14] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:01.827905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:01.860744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4dd27-5cfb-41f1-8455-b9f969cd593c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:01.863039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:01.897824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:01.898725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.899006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.899276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.908304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:01.926655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecabf67-76d4-4db8-842c-1be6269bef9a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:01.959650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:01.960338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:01.961217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:01.961472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:01.962172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:01.962475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:01.963872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:01.967055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:01.968918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:01.975484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 8/15] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:01.981890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:02.068096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 8/16] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 10:47:02.073567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:02.076237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:02.077923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546986d-a03e-434d-863c-e63d97c2c9bc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:02.112084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:02.112714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:02.112992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:02.113575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:02.114412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:02.115858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:02.117626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 9/17] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:01 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:02.122828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:02.138670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:02.172372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e682f4-8fe9-454e-a481-2ca9018a7b1a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:02.204660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:02.205459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:02.205740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:02.206036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:02.221512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7229b254-4b2e-4f7e-9cc2-5b45d01f4780 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:02.224442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:02.225917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:47:02.244688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:02.245619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:47:02.246628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:47:02.246628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:47:02.246628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:02.246628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 9/18] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:02.251641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:02.259363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:02.260003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:02.260284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:02.260561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:02.261686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:02.263191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:02.264906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 10/19] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:02.270783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:02.309384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:02.351275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:02.355741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed4ac8f-ad50-4e78-8b96-d64e333a82c8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:02.371700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c10a2-c11d-4a20-9af2-1745647699d3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:02.388790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:02.389402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:02.389688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:02.389995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:02.390652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:02.390959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:02.392347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7df68e9-7cce-4568-b10e-cf56c453ad1a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:02.405000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:02.405608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:02.405883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:02.406184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:02.406897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.domain.id=default, user.id=79c132c0cbf74b0899faa3036130f049) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:02.407198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:02.408608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453959b-634f-4bd0-b951-99277ac2fdcb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:02.409932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 10/20] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:47:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:02.415435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 11/21] 10.222.0.46 () {56 vars in 1322 bytes} [Fri Oct 28 10:47:02 2022] GET /identity/v3/role_assignments?user.id=79c132c0cbf74b0899faa3036130f049&scope.domain.id=default&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 233 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:04.974707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f076a09-4a81-4389-af3b-7b02df4ec07e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:04.975614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f076a09-4a81-4389-af3b-7b02df4ec07e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:04.976189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f076a09-4a81-4389-af3b-7b02df4ec07e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:04.977736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 11/22] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:04.983641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140c072-ddc0-419f-9ff6-5c855ca6769f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:04.984433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140c072-ddc0-419f-9ff6-5c855ca6769f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:04.984898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140c072-ddc0-419f-9ff6-5c855ca6769f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.007425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5140c072-ddc0-419f-9ff6-5c855ca6769f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:05.009448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5140c072-ddc0-419f-9ff6-5c855ca6769f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:05.052961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3ae383-cfe4-48a8-bca5-007b2f0438ec None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.053662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3ae383-cfe4-48a8-bca5-007b2f0438ec None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.054191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3ae383-cfe4-48a8-bca5-007b2f0438ec None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.059792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 12/23] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:05.086370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31260ac0-64e5-4f06-8733-d71799422952 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.086964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31260ac0-64e5-4f06-8733-d71799422952 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.087432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31260ac0-64e5-4f06-8733-d71799422952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.096915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 12/24] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:05.122992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-31260ac0-64e5-4f06-8733-d71799422952 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:05.124542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-31260ac0-64e5-4f06-8733-d71799422952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:05.223557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 13/25] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:05.394813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f3e837-b1e6-4c6b-87f4-14cc674eaf04 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.395176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f3e837-b1e6-4c6b-87f4-14cc674eaf04 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.395374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f3e837-b1e6-4c6b-87f4-14cc674eaf04 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.396879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 13/26] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:05.408402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.408690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.408970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.435072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:05.436567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:05.486275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:05.520237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 14/27] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:05.572233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f3e837-b1e6-4c6b-87f4-14cc674eaf04 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:05.682822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:05.704275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:05.790479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f3e837-b1e6-4c6b-87f4-14cc674eaf04 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:05.846199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a88daf-2493-428d-a2ee-586e2d0e688f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:05.860771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b776977-0340-4493-b064-e2529787f13f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:05.861413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b776977-0340-4493-b064-e2529787f13f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.861690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b776977-0340-4493-b064-e2529787f13f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.861959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b776977-0340-4493-b064-e2529787f13f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.879245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b776977-0340-4493-b064-e2529787f13f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:05.880796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b776977-0340-4493-b064-e2529787f13f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7b776977-0340-4493-b064-e2529787f13f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:47:05.911842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:05.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:47:05.913516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:47:05.913516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:47:05.913516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:05.918097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 14/28] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:05.919924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20dc6af-5e19-4beb-86bc-692334fda3b8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.920363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20dc6af-5e19-4beb-86bc-692334fda3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.920653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20dc6af-5e19-4beb-86bc-692334fda3b8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.921954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 15/29] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:05.922792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20dc6af-5e19-4beb-86bc-692334fda3b8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:05.925353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:05.926231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:05.926514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:05.926908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:05.927593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:05.928992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:06.038129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20dc6af-5e19-4beb-86bc-692334fda3b8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.069443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.079022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 15/30] 10.222.0.46 () {60 vars in 1131 bytes} [Fri Oct 28 10:47:05 2022] POST /identity/v3/roles => generated 219 bytes in 392 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:47:06.080787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.080961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:06.086322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32236524-a2af-462f-a4fb-c8b71f451ecd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:06.109210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 16/31] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:47:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:06.110823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.111117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.111395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.134917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:06.136730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:06.160180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.252567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 17/32] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:06.253339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.286157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76414f2-e190-435b-b622-7a5f29c2ea77 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.329878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.330625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.330911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.331189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.366787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:06.380857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:06.381739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:06.382530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:06.383640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 16/33] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:06.388746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.476139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e9eeb2-a8e5-43bb-b2c9-4c9c13627ef8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.506910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.530948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.535596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.536607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.536889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.537162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.537820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:06.538810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0c8423-b02b-49af-a06f-19957f8bff67 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.539751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc19c6b-eb1a-452d-b931-f76ac4cafa66 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:06.582754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.583762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.584356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.584841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.585905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:06.586477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:06.588427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f02562-eb91-4cf2-9b95-7bfce992cbc0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:06.601751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 17/34] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:06.603201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b402c3c-94d1-4c11-94d8-3369250e004a None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.603601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b402c3c-94d1-4c11-94d8-3369250e004a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.604633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b402c3c-94d1-4c11-94d8-3369250e004a None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.605233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 18/35] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:06.606102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fbe0-d162-451f-a1de-2e5574acc955 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.606370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fbe0-d162-451f-a1de-2e5574acc955 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.607228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4fbe0-d162-451f-a1de-2e5574acc955 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.611998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 19/36] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:06.613075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.613360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.613629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.633366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:06.634806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:06.688132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 18/37] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:47:06 2022] POST /identity/v3/projects => generated 301 bytes in 435 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:06.689571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c310b-5572-445c-b15e-46041a975622 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.689860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c310b-5572-445c-b15e-46041a975622 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.690166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c310b-5572-445c-b15e-46041a975622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.705874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 20/38] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:06.711004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-517c310b-5572-445c-b15e-46041a975622 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:06.711187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.713004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-517c310b-5572-445c-b15e-46041a975622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:06.791344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 19/39] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:06.792481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c310b-5572-445c-b15e-46041a975622 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.855822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.885431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c1b37-8436-4e77-8283-ac2e9b407265 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.925997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.926635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.926967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.927248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.928327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:06.929050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:06.930632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:06.932188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 21/40] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:06 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:06.934367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c310b-5572-445c-b15e-46041a975622 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:06.938966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:06.986156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:06.987131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:06.987637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:06.988269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:06.989142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:06.991720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.011564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 20/41] 10.222.0.46 () {60 vars in 1131 bytes} [Fri Oct 28 10:47:06 2022] POST /identity/v3/roles => generated 225 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:07.012634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:07.089907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0f1ad-a000-4aa1-885a-ea69760eef12 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:07.135428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:07.136158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:07.136392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:07.136667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:07.137896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:07.139468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.141426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 22/42] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:07.142041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:07.173744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:07.249235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a35c8-0b73-4f9b-9e52-39d377caeb41 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:07.272056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91742f4-7381-47ba-a3fd-3dabdbf4ea99 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:07.290125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:07.290773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:07.291056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:07.291324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:07.292505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:07.293986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.295579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 21/43] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:07.296321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:07.307998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:07.308917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:07.309202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:07.309482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:07.310187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:07.311556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.324254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:07.325795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 23/44] 10.222.0.46 () {60 vars in 1131 bytes} [Fri Oct 28 10:47:07 2022] POST /identity/v3/roles => generated 223 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:07.326748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:07.418293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08de8036-bbc3-458a-9ef1-bff61a0efb6d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:07.423399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:07.444006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8729f8-4abe-407b-a407-64c3f7f8939d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:07.457317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:07.458032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:07.458474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:07.458715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] PATH_INFO: `/v3/domains/default/users/79c132c0cbf74b0899faa3036130f049/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:07.461292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=79c132c0cbf74b0899faa3036130f049, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:07.463008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a91f58-d7d4-4f88-b95e-bd5e6c7cb078 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.480408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 22/45] 10.222.0.46 () {58 vars in 1361 bytes} [Fri Oct 28 10:47:07 2022] PUT /identity/v3/domains/default/users/79c132c0cbf74b0899faa3036130f049/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:07.486744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:07.487400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:07.487684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:07.487963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:07.488649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:07.490279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97a04d6-2369-4f82-82f8-0a2c033242a1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:07.527314 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 24/46] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:47:07 2022] POST /identity/v3/projects => generated 312 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:09.631434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec897440-ae80-4c95-9be8-b9251aa21de0 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:09.631975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec897440-ae80-4c95-9be8-b9251aa21de0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:09.631975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec897440-ae80-4c95-9be8-b9251aa21de0 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:09.633920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 23/47] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:09.639193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:09.639575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:09.639760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:09.657825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:09.659034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:09.689558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9db83-09e9-463a-a337-3eadddee671b None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:09.689846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9db83-09e9-463a-a337-3eadddee671b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:09.690166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9db83-09e9-463a-a337-3eadddee671b None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:09.691493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 24/48] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:09.696895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:09.697180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:09.697457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:09.714916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:09.716120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:09.727723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 25/49] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:09.781942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 25/50] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:09.996794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.052032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.061279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:10.071148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:10.096501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b6702-8696-4a83-9d2e-9e104c455b2e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.131907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.132569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.132853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.133134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.143020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:10.148029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:10.149010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:10.149916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 26/51] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:10.151697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234bf42-d87e-43c6-876b-3c0d464f9080 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.156502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.185476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.186136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.186419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.186690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.187792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.189244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.191180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 26/52] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:10.196385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.257132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d568d2-15f1-43fb-a1ac-e29f068f33e8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.282293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:10.300010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff02a71f-1eff-4f05-aa49-6f5ce244cbf7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.336112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.336744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.337026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.337302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.338006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.339504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.368168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 27/53] 10.222.0.46 () {60 vars in 1140 bytes} [Fri Oct 28 10:47:10 2022] POST /identity/v3/projects => generated 298 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:47:10.386860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:10.396436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.397075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.397360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.397639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.398897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:10.399206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.400647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.410160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 27/54] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:10.416747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.524205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc992795-901e-401c-8519-906f93d01a9d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.559171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.559806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.560085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.560356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.561491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.562987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.564922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 28/55] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:10 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:10.570964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.666844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48db8fa-1ddc-4d7d-a9f8-514e55a08e2e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.699413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.700038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.700315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.700582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.701677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.703149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175e130-ffcb-4cd6-8aa3-8c90232a9f44 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.704858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 28/56] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:10.710692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:10.714611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:10.810450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87773c81-ef7a-4fdd-8353-ee1ecdd6b6d4 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:10.842303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:10.842908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:10.843188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:10.843460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:10.844182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.domain.id=default, user.id=79c132c0cbf74b0899faa3036130f049) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:10.844492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:10.845921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059d0ba1-e632-44de-a31d-b6f7fac91256 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:10.852363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 29/57] 10.222.0.46 () {56 vars in 1322 bytes} [Fri Oct 28 10:47:10 2022] GET /identity/v3/role_assignments?user.id=79c132c0cbf74b0899faa3036130f049&scope.domain.id=default&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 530 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:12.519561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f780603-78df-4431-8b81-4067590aab20 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:12.520067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f780603-78df-4431-8b81-4067590aab20 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:12.520151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f780603-78df-4431-8b81-4067590aab20 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:12.521483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 29/58] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:12.526540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:12.526969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:12.527100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:12.545231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:12.546491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:12.606785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:12.612546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 30/59] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:12.878325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f780603-78df-4431-8b81-4067590aab20 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:12.934306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:12.975848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f780603-78df-4431-8b81-4067590aab20 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:13.008621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:13.009238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:13.009536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:13.009791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:13.010916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:13.012349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc44d43-cc95-479f-ae71-c5418ef7372f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:13.013878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 30/60] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:13.019308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:13.114929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c8b036-7229-4aa7-b624-b0cef585b7f0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:13.147814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:13.148393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:13.148675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:13.148951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:13.149607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:13.151078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7158d1a-60cd-40fa-aae6-fd6ba29a26be None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:13.176512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 31/61] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:13 2022] POST /identity/v3/users => generated 275 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:16.429211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b200f5ed-5180-4a29-8251-97b810a1faff None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.430190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b200f5ed-5180-4a29-8251-97b810a1faff None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.431211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b200f5ed-5180-4a29-8251-97b810a1faff None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.432779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 31/62] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:16.438227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9142a38-c42e-4533-821d-b7c5dd6a39cb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.439050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9142a38-c42e-4533-821d-b7c5dd6a39cb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.439530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9142a38-c42e-4533-821d-b7c5dd6a39cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.463108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a9142a38-c42e-4533-821d-b7c5dd6a39cb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:16.464328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a9142a38-c42e-4533-821d-b7c5dd6a39cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:16.535213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:16.560250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 32/63] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:16.744974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dc456-1cec-4535-8251-de349b6e9f67 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.745658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dc456-1cec-4535-8251-de349b6e9f67 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.746174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dc456-1cec-4535-8251-de349b6e9f67 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.747736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 32/64] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:16.753305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65458030-a970-4ef6-8309-53d0f9c84f8c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.753598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65458030-a970-4ef6-8309-53d0f9c84f8c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.753878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65458030-a970-4ef6-8309-53d0f9c84f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.783012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-65458030-a970-4ef6-8309-53d0f9c84f8c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:16.784229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-65458030-a970-4ef6-8309-53d0f9c84f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:16.816493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f49afa2-e777-48be-820c-8e1464c7f9a4 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.817084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f49afa2-e777-48be-820c-8e1464c7f9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.817558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f49afa2-e777-48be-820c-8e1464c7f9a4 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.819358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 33/65] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:16.824681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.825198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.825694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.846356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:16.848173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:16.872112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 33/66] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:16.874325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d75e45-39c5-42bd-80b4-681464ba0468 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.874794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d75e45-39c5-42bd-80b4-681464ba0468 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.875240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d75e45-39c5-42bd-80b4-681464ba0468 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.876647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 34/67] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:16.883284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:16.883633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:16.883916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:16.902280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:16.903691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:16.941169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 34/68] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:17.004474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 35/69] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:17.027334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.105859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.173671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dec188-215a-4661-898b-9639a59f2181 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.177668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.218104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.218751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.219034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.219310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.230841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b5eaa1ac-0f8b-4e8e-9a41-d32d8006932b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:17.235760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:17.236730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:17.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:17.239897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 35/70] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:17.241657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea1b1e-91b5-4a08-88f3-df3100165460 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.242153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea1b1e-91b5-4a08-88f3-df3100165460 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.242592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea1b1e-91b5-4a08-88f3-df3100165460 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.244318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 36/71] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:17.245959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea1b1e-91b5-4a08-88f3-df3100165460 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.246653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.310052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d67fbd-48bf-46d8-83b7-93d82ff782d9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.345313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.346002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.346259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.346530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.350454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea1b1e-91b5-4a08-88f3-df3100165460 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.357150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-0a0ef771-d3bd-4f98-87c0-d178cdf411bc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:17.361912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:17.362751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:17.363500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 36/72] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:17.364589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160229d-6297-4cb2-8613-0051557624fb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.364837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160229d-6297-4cb2-8613-0051557624fb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.365112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160229d-6297-4cb2-8613-0051557624fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.368643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.382782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d160229d-6297-4cb2-8613-0051557624fb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:17.384007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d160229d-6297-4cb2-8613-0051557624fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:17.389514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.390148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.390429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.390703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.391374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:17.391672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:17.393047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:17.401067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 37/73] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:17.402004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.452515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 37/74] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:17.453355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d160229d-6297-4cb2-8613-0051557624fb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.491384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.505559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f139ffe-7ca1-407a-aaf0-8e3dd23ecd4b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.538614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.539244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.539524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.540359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.550672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:17.556532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:17.557332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:17.558104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 38/75] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:17.558730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.573167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d160229d-6297-4cb2-8613-0051557624fb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.597146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.610855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.611429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.611713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.611985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.622405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:17.627725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:17.629369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 38/76] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:17.630914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.675421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83aa48-771a-42cf-8645-4d34e819344b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.712500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.713113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.713393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.713664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.714771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:17.715079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:17.716808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:17.725610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 39/77] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:17.726351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.733062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.739743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe973-7ecd-4b39-aabb-ecb4e6fe2832 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.773174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.773834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.774171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.774441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.775607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:17.777056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:17.778653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 39/78] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:17.779396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.829005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.832955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214f85b8-53bf-4fb0-a676-1408d1deeddd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.866486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.867023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.867304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.867574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.868291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:17.868595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:17.870003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:17.877552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a0983-d5f9-432e-9ab9-0d7dabbc2703 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:17.877666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 40/79] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:17.878529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.911018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:17.911630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:17.911910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:17.912184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:17.912836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:17.913139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:17.914571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:17.924245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 40/80] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:17.925017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:17.948976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:17.981584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e781e1-a978-4668-9a91-081c2d631088 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.013728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.014381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.014660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.014932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.016040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.017479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.019049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 41/81] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:18.019809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.022854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c5a56-374b-48d0-899f-18eed8805520 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.048085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.058148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.058784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.059062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.059334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.072753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.074234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.075822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 41/82] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:17 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:18.076635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.122691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c5a9c-866c-4b15-b186-2a831b2edad6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.156472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.157072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.157355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.157625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.158590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.177879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264401b7-5017-4a1b-ab3e-2e27c3ff3ec5 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.211026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.212254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.212535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.212810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.213931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.215390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.216939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 42/83] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:18.217689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.226183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.228124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.254755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.319960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c8690-47a2-40dc-a8de-05486ed4e671 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.327467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 42/84] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:18.328291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.358115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.358744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.359022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.359292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.360396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.361820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.363408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 43/85] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:18.364365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.383955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.426970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed379-d3e4-4f25-95cd-356556687bd0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.453248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.461109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.461727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.462031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.462311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.462965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.463728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b93f54-6a3a-4dee-88b9-b3e474b2f7c3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.464411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.491912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 43/86] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:18 2022] POST /identity/v3/users => generated 289 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:18.493236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.509175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.509813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.510130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.510400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.605174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.606866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.609179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc32a26-3eac-4d7d-80a0-66231f969b45 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.628905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.648800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.649388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.649679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.649958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.714828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.727673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 44/87] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:18.728975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.735653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.737140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.849965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c737ff-1adc-4b9d-b6af-69a460af642f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:18.857251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 44/88] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:18.862467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.875210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.896147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:18.896904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:18.897185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:18.897585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:18.898746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:18.900453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:18.902366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 45/89] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:18.902901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:18.963714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:18.981958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217e41c-cdd0-4a01-8d0e-bdabf650d3f3 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.028666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.030060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.030387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.030670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.031791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4c48c8-62a4-4eaa-a5c4-369ae40525c0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.069675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.070616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.071542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.071911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.073138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.074711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.076511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 46/90] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.077384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.120039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.122102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.143549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:19.203630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dc9d7f-8dd8-4256-8a41-f7d8ef1f60eb None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.212695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:19.231495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 45/91] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:19.232479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.255617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.256467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.256931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.257299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.259295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.260955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.262847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 47/92] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.263701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.360671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f673be-8393-4e75-b880-675d6fa90338 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.391125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f14e1d-b40e-4ed4-8465-1c5f9ba28709 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.397711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.398611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.399115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.399579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.400512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=79c132c0cbf74b0899faa3036130f049) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:19.401106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.402775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.412000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 46/93] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/role_assignments?user.id=79c132c0cbf74b0899faa3036130f049&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.413127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.433288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:19.456157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.457279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.457849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.458349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.459406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:19.466004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.468651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.474460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:19.488344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 48/94] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=770c965de4374724a8a6de8042997c97 => generated 259 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.490911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.652365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418835d-0eb4-4b9c-97dc-27f51b4b8727 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.687698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9af728a-770a-4620-8e23-cf61dc913f55 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.733427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.734702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.735716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.736571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.737090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.738076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:19.738968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.739450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.739889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.740299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] PATH_INFO: `/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.742026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3c13e-7760-4c2e-8b7a-480a6b505564 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.745609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] RBAC: Authorizing `identity:get_project(project_id=0ed5f0d872ce4a048e14eeb0588e0569)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.747341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.749312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 47/95] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569 => generated 312 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.754263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 49/96] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=1126c16f80da4dde8de7192a7858171e => generated 259 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:19.764629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:19.910142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceeefe-b151-485a-ae33-131bd7f7cd69 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:19.928172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:19.952874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:19.953764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:19.954235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:19.954646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:19.955514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=0ed5f0d872ce4a048e14eeb0588e0569, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:19.956316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:19.957967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f07b8b-4b84-476c-a959-2449032c43e2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:19.966582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 48/97] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:19 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=0ed5f0d872ce4a048e14eeb0588e0569&role.id=770c965de4374724a8a6de8042997c97 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:21.100786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c829ca-8de8-43a6-9056-dfeaad177b14 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:21.101307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c829ca-8de8-43a6-9056-dfeaad177b14 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:21.101307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c829ca-8de8-43a6-9056-dfeaad177b14 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:21.103737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 50/98] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:21.112234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be720-3574-4441-8ed8-dd051803fdb9 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:21.113026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be720-3574-4441-8ed8-dd051803fdb9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:21.113456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be720-3574-4441-8ed8-dd051803fdb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:21.138965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-477be720-3574-4441-8ed8-dd051803fdb9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:21.140470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-477be720-3574-4441-8ed8-dd051803fdb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:21.231425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 49/99] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:21.499969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fb535-2982-42b9-a9d5-d2cd593438e1 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:21.500173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fb535-2982-42b9-a9d5-d2cd593438e1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:21.500608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fb535-2982-42b9-a9d5-d2cd593438e1 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:21.502114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 51/100] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:21.507656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:21.508162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:21.508626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:21.532631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:21.534057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:21.611245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 50/101] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:21.688479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7fb535-2982-42b9-a9d5-d2cd593438e1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:21.756887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:21.819985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7fb535-2982-42b9-a9d5-d2cd593438e1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:21.866457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:21.867528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:21.868431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:21.868709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:21.879148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:21.888512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:21.889352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:21.890114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 52/102] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:21.899711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:21.966742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:22.045957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be51ed0-0268-4b46-aeb9-d5c859386789 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:22.080867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:22.081639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.082085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.082369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.083042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:22.083348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:22.084729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:22.093932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 51/103] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:22.099478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:22.144086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:22.237041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b95b54-bb92-4a9d-83f0-84573267e586 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:22.272065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:22.305932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:22.307156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.307646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.308137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] PATH_INFO: `/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.337854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] RBAC: Authorizing `identity:get_user(user_id=73d6726ac7a8484fbea9e4bc7bc3ac01)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:22.339857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:22.346743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 53/104] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:22 2022] GET /identity/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01 => generated 289 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:22.347990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:22.399502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb75cf3-a9cb-4791-a71c-2719cf98d749 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:22.400975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:22.490724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:22.491394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.491679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.491959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.493744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:22.495231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:22.496971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 52/105] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:22.502589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:22.519939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9c0a72-d6b6-42d1-8b2e-6c87b67a1947 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:22.582727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:22.583361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.583642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.583918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.648326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:22.684661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6690933-74b1-4f9b-ad61-741dacf3bcea None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:22.712556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:22.714055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:22.749679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:22.750345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.750626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.750901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.751595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:22.753112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1786c574-30f0-4755-bee8-6f7e6dabe736 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:22.819965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 53/106] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:47:22 2022] POST /identity/v3/projects => generated 302 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:22.822151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b156ac-1770-4e07-b84f-44a39c2f4adb None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.822892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b156ac-1770-4e07-b84f-44a39c2f4adb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.823420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b156ac-1770-4e07-b84f-44a39c2f4adb None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.825005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 54/107] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:22.841835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5226b85-51c0-4fd6-bd4e-efef9635960e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:22.842756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5226b85-51c0-4fd6-bd4e-efef9635960e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:22.843265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5226b85-51c0-4fd6-bd4e-efef9635960e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:22.892929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a5226b85-51c0-4fd6-bd4e-efef9635960e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:22.895413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a5226b85-51c0-4fd6-bd4e-efef9635960e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:22.905028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 54/108] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:22.912992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:22.938457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.025130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.026824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 55/109] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:23.028385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713546ca-bc14-4ba3-b852-f44880642165 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.028890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713546ca-bc14-4ba3-b852-f44880642165 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.029364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713546ca-bc14-4ba3-b852-f44880642165 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.030926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 56/110] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:23.032196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823335f-e038-45f6-8cef-c50faf4521b1 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.032780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823335f-e038-45f6-8cef-c50faf4521b1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.033254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823335f-e038-45f6-8cef-c50faf4521b1 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.034739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 57/111] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:23.036507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.037010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.037522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.075781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a5b4e-6c19-44d3-8378-e1124f8ed45b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:23.087065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:23.088294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:23.119235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:23.120113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.120553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.121471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.123088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:23.126048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366daf1-768d-4142-8b5e-0fe433f8059a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:23.128805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 55/112] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:22 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:23.130113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.130544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.130958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.154232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:23.156022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:23.186002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 58/113] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:23.191205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.260396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 56/114] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:23.354541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.361932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.377610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842dc43f-7898-4f9d-8233-562d0c635705 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:23.434320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:23.435284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.435757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.436243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.437160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=73d6726ac7a8484fbea9e4bc7bc3ac01) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:23.437677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:23.439331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984dc4b5-2862-4ee8-bd47-e7120bdfb8f7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:23.445429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 59/115] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/role_assignments?user.id=73d6726ac7a8484fbea9e4bc7bc3ac01&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 259 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:23.446974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1139f059-9beb-4f64-9664-100c15a77b14 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.447456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1139f059-9beb-4f64-9664-100c15a77b14 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.447918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1139f059-9beb-4f64-9664-100c15a77b14 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.449431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 60/116] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:23.455724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.456353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.457060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.485236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.494540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:23.501381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:23.551928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85862167-919c-4ca9-91db-dc324eff1e48 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:23.605882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 61/117] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:23.607324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.658817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:23.659421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.659682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.659978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.698836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.701261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:23.707337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:23.708178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:23.708908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 57/118] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:23.709532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.736266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e743a9-d020-494e-b7ae-3fdc799757db None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:23.804981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:23.806184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.806469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.806723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.833061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:23.838117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:23.838955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:23.839685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 62/119] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:23.840308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.873346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167a7423-5cdd-466d-918a-96167a3802e4 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:23.901318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:23.916393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:23.916971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:23.917256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:23.917536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:23.932721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:23.939704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:23.940554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:23.941317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 58/120] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:23.941921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:23.993269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f4a34-f065-4713-9804-0792bec04dcf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.036774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.038327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.038840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.039312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.041195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:24.042203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.044145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.046992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.057275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 63/121] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.058374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.070248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14056c9d-cfff-45d7-a506-7c95403b38bf None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.119708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.120998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.121485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.122300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.123778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:24.124418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.127009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.143854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 59/122] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.143854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.180846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365fa6dd-ae82-4d6f-82f0-8bbc31361fc7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.192213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.225795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.227280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.227767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.228245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.242213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:24.246432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435cdd84-a596-481d-aa38-d830ded8acb1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:24.248767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:24.249660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:24.250477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:24.256301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 64/123] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:24.257481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.293386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.294666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.295153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.295629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.296496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:24.297001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.298842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.309737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 60/124] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.310962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.330660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.381472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39699c3-9069-4297-89c7-7e3d93e4455c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.423847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.424495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.424784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.425067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.425677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05054c6a-d14d-435b-a5e0-1a7deeb42ca7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.426252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.428030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.430585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 65/125] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.431421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.438864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.466249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.467406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.467890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.468423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.470011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.471787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.479149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 61/126] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.480253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.554871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e205ddb-7b90-4cba-b1b8-5d0ef9bdbd0d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.585119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.595944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523959bd-cefe-4d3d-bc3d-5c4a4df5cdb9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.597426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.598168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.598540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.598884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.600231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:24.600231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.601367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.614572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 66/127] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.614572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.647292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.648333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.649068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.649762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.651117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.653661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.655473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 62/128] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.656393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.711177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.738057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bfeda5-ad60-4e7b-b855-5543357f93b5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.771247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1faf967e-2707-49e9-9b11-1003fcc795fc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.777830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.778736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.779215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.779708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.781019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.782811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.784732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 67/129] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.785664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.817636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f0c00-580c-4687-8803-35877b258573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.818297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f0c00-580c-4687-8803-35877b258573 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.818577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f0c00-580c-4687-8803-35877b258573 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.818851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233f0c00-580c-4687-8803-35877b258573 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.820112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233f0c00-580c-4687-8803-35877b258573 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.821837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233f0c00-580c-4687-8803-35877b258573 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.823815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 63/130] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.824490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f0c00-580c-4687-8803-35877b258573 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.856830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.911454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65846d-d856-437e-8908-d958f4a1593d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.933577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233f0c00-580c-4687-8803-35877b258573 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:24.948822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:24.955264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.957286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.957896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.958444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.959839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.962178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.964655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 68/131] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.969242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:24.977965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:24.979163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:24.980072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:24.980574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:24.982540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:24.986102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:24.988839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 64/132] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:24 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:24.989800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:25.081375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55e53b-ccc6-428c-a826-ee8cd0599b5a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:25.106960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:25.119131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:25.119838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.120126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.120402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/79c132c0cbf74b0899faa3036130f049/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.123259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, user_id=79c132c0cbf74b0899faa3036130f049, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:25.124834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:25.132286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e31680-b16e-4b64-870e-ccdbba126e63 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:25.141497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 69/133] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:24 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/79c132c0cbf74b0899faa3036130f049/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:25.142781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:25.171324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:25.172126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.172577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.172934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/7d855e0bceeb417d9bfcb6751dd67251/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.182483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, user_id=7d855e0bceeb417d9bfcb6751dd67251, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:25.184104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:25.229201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 65/134] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:24 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/7d855e0bceeb417d9bfcb6751dd67251/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:25.229968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:25.290803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180e7a-7350-4edf-9238-716286d440fc None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:25.330378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:25.331394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.332376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] PATH_INFO: `/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.333868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] RBAC: Authorizing `identity:get_project(project_id=0ed5f0d872ce4a048e14eeb0588e0569)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:25.335742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:25.338846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 70/135] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:25 2022] GET /identity/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569 => generated 312 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:25.344443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:25.346789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:25.411709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:25.480151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8f66a-3d49-482d-835c-513391c1aaa0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:25.481356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5a3edf-db2e-4e4b-9edb-ec1c37c22962 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:25.520956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:25.522129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.522826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.523293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] PATH_INFO: `/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569/users/7d855e0bceeb417d9bfcb6751dd67251/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.525718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] RBAC: Authorizing `identity:create_grant(project_id=0ed5f0d872ce4a048e14eeb0588e0569, user_id=7d855e0bceeb417d9bfcb6751dd67251, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:25.527478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f80cd6-9198-42b4-ace0-c0e0c50dd051 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:25.552352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 71/136] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:25 2022] PUT /identity/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569/users/7d855e0bceeb417d9bfcb6751dd67251/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 10:47:25.565733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:25.566818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.567285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.567779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/7d855e0bceeb417d9bfcb6751dd67251/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.570384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, user_id=7d855e0bceeb417d9bfcb6751dd67251, role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:25.572840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b5743a-7028-4e6f-9b11-c8fe5eb0865e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:25.593416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 66/137] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:25 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/7d855e0bceeb417d9bfcb6751dd67251/roles/1126c16f80da4dde8de7192a7858171e => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:25.779628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37591f-97e5-4cf8-8cb9-09cb0cefbecd None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.779921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37591f-97e5-4cf8-8cb9-09cb0cefbecd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.780102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e37591f-97e5-4cf8-8cb9-09cb0cefbecd None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.781577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 72/138] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:25.787550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c94e8-fdbb-4789-a2b9-173084bad9dd None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:25.788169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c94e8-fdbb-4789-a2b9-173084bad9dd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:25.788645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7c94e8-fdbb-4789-a2b9-173084bad9dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:25.809504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8f7c94e8-fdbb-4789-a2b9-173084bad9dd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:25.811136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8f7c94e8-fdbb-4789-a2b9-173084bad9dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:25.889151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 67/139] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:26.121126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f5855-8b25-44a9-8b28-2105b336a8a8 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:26.121126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f5855-8b25-44a9-8b28-2105b336a8a8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:26.121126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f5855-8b25-44a9-8b28-2105b336a8a8 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:26.121126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 73/140] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:26.127076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:26.127610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:26.128088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:26.159023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:26.160602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:26.203042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:26.230556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f5855-8b25-44a9-8b28-2105b336a8a8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:26.323390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 68/141] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:26.496620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f5855-8b25-44a9-8b28-2105b336a8a8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:26.518355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:26.564222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:26.565230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:26.565511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:26.565960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:26.568581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:26.570035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:26.571652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 74/142] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:26.579051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:26.740835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4662c4f-1efb-4624-a1dc-1028a0fdbfc2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:26.794765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:26.795730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:26.796214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:26.796689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:26.797560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:26.798028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:26.799074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:26.832904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 69/143] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:26 2022] POST /identity/v3/users => generated 283 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:26.946711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0550bd-d617-4ba4-98cd-aebe048b1694 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:27.002253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:27.002905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:27.003183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:27.003454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:27.024270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:27.030007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:27.030878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:27.031612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 75/144] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:27.039169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:27.070991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:27.185174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd35f7e-96f1-439b-bfbc-4b066c7398a1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:27.224958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:27.225601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:27.225946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:27.226244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:27.227473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:27.227779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:27.229462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:27.242541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 70/145] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:27.249666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:27.306285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:27.376854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b44cf57-37d8-4a6f-9990-735af0c949b4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:27.414058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:27.415372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:27.415654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:27.415927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] PATH_INFO: `/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:27.417067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] RBAC: Authorizing `identity:get_user(user_id=73d6726ac7a8484fbea9e4bc7bc3ac01)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:27.418612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:27.420492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 76/146] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:27 2022] GET /identity/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01 => generated 289 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:27.425960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:27.526094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:27.546535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96e4060-81c1-46a1-bd76-6566770f6e66 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:27.584855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:27.585573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:27.585758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:27.586049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:27.587708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:27.589157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106be23-3d2b-4565-b1f7-fb044295d2e7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:27.590864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 71/147] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:27 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:27.596963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:27.711621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3edb19-5692-41b1-b65f-ea59542f5211 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:27.723169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:27.752426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:27.753093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:27.753369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:27.753640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/73d6726ac7a8484fbea9e4bc7bc3ac01/roles/a6d2f4a8b54a44b29b129335ac106b7b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:27.755673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, user_id=73d6726ac7a8484fbea9e4bc7bc3ac01, role_id=a6d2f4a8b54a44b29b129335ac106b7b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:27.757178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979d596-610d-4506-aa91-2d2d7f4ebab9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:27.783003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 77/148] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:27 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/users/73d6726ac7a8484fbea9e4bc7bc3ac01/roles/a6d2f4a8b54a44b29b129335ac106b7b => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:28.541800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717cf239-69e7-4374-afde-5fce88b05f00 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:28.542169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717cf239-69e7-4374-afde-5fce88b05f00 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:28.542345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717cf239-69e7-4374-afde-5fce88b05f00 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:28.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 72/149] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:28.549370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e9-4ce5-47ef-978c-a5d2266cf4e3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:28.549659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e9-4ce5-47ef-978c-a5d2266cf4e3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:28.549935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499176e9-4ce5-47ef-978c-a5d2266cf4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:28.572013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-499176e9-4ce5-47ef-978c-a5d2266cf4e3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:28.573532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-499176e9-4ce5-47ef-978c-a5d2266cf4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:28.652399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 78/150] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:28.931814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1083db7-a75e-4c16-bcf0-f54dcac216de None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:28.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1083db7-a75e-4c16-bcf0-f54dcac216de None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:28.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1083db7-a75e-4c16-bcf0-f54dcac216de None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:28.933949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 73/151] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:28.939346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb3628-16f3-4861-9cf8-bdb275a399fa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:28.939902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb3628-16f3-4861-9cf8-bdb275a399fa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:28.940384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bb3628-16f3-4861-9cf8-bdb275a399fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:28.963829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c6bb3628-16f3-4861-9cf8-bdb275a399fa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:28.965367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c6bb3628-16f3-4861-9cf8-bdb275a399fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:29.054046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 79/152] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:29.281507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1083db7-a75e-4c16-bcf0-f54dcac216de None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:29.375894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fba17a-9a9f-4c14-b91f-584ad72ee7cb None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.377039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fba17a-9a9f-4c14-b91f-584ad72ee7cb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.377549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fba17a-9a9f-4c14-b91f-584ad72ee7cb None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.379397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 80/153] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:29.384779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.385325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.385817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.419370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:29.420558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:29.443896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1083db7-a75e-4c16-bcf0-f54dcac216de None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:29.473426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:29.504119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:29.504759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.505034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.505303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.536573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d8d78b07-cd7a-4b23-9a1e-e089bb44ac0e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:29.541954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:29.542859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:29.543641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:29.552381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 74/154] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:29.556608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1fc4b7-2c89-4aed-8aaf-9ca46bce5d8d None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.556883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1fc4b7-2c89-4aed-8aaf-9ca46bce5d8d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.557151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1fc4b7-2c89-4aed-8aaf-9ca46bce5d8d None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.564729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 75/155] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:29 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:29.565524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1fc4b7-2c89-4aed-8aaf-9ca46bce5d8d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:29.581531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 81/156] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:29.582446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:29.592435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:29.688094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e930d8-9de1-4669-b80f-a910ca71e19c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:29.723750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:29.724387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.724660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.724927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.736550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3a5b3adf-ed74-405d-a325-87fa960cefa6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:29.741750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:29.742677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:29.743646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 82/157] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:29.744297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.744366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.744941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.767436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:29.768614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:29.784139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:29.803775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1fc4b7-2c89-4aed-8aaf-9ca46bce5d8d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:29.842777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 83/158] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:29.843460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:29.890713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:29.891424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:29.891716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:29.891990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:29.892674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:29.892978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:29.894421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:29.911874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 76/159] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:29.917927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:29.981099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035764e3-2cf6-4751-8c5a-4a11f79048e6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.021107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.021725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.022018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.022294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.022962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:30.023259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.024765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:30.033324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 84/160] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:30.038205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.051787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.084693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.145559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35979ad6-580e-486a-a1fc-30b7fe4d2dff None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.176352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca6843-2fb8-4743-8321-a6c3440ac74c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.182264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.182903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.183182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.183454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.194305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:30.199628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:30.200538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:30.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 85/161] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:30.202001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.290335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.311437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be9605c-b37a-40d7-992e-9789ee7a2379 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.314698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.315363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.315647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.315925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] PATH_INFO: `/v3/users/79c132c0cbf74b0899faa3036130f049` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.322686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] RBAC: Authorizing `identity:get_user(user_id=79c132c0cbf74b0899faa3036130f049)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.324172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:30.325760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 77/162] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:29 2022] GET /identity/v3/users/79c132c0cbf74b0899faa3036130f049 => generated 247 bytes in 408 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:30.326561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.372779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.374392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.376973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.377466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.379490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.381114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:30.387452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 86/163] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:30.388419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.529899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbb7da5-47ca-4342-b87d-8d1bf269babb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.544736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc57752-1a74-440c-a882-240a296c214a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.567153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.568382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.568886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.569331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.578277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.652080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.652618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.652901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.653176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.653857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:30.654186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.655580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:30.666768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.673192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 78/164] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:30.674026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.695034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.697457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:30.820758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a791a6-7d36-420a-8681-00d3fc94d71d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:30.868850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:30.869466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:30.869739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:30.870022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:30.920448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 87/165] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:30.921250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:30.995144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:30.998075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:30.999542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.002755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:31.106513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9722f6d-97be-4f01-9529-d88cc57a9b42 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.183423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.183990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.184270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.184544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.204279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 79/166] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 530 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:31.204917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.218583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:31.225094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:31.226234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 88/167] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:31.227128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.369092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:31.372862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725f59b-9fd1-41ba-a896-1317345fb5de None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.427461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.428348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.428766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.429210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.431089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:31.432704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.434875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 80/168] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:31.435791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.473123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:31.479424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceba72-f01b-40ab-b906-216d617fca81 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.548621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.549327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.549615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.549890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.554694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:31.556766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.558809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 89/169] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:31.559590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.611298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e5add3-d6ff-4e24-ac8f-ef1c9c028fcd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.646808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.647450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.647724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.647992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.649217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:31.650801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.652499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 81/170] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:31.653255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.702424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:31.762441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67317b39-9fe1-4d91-b344-499786186636 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.789074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9cfa3-3bd1-41d3-b5e1-76c17b0bbbcd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:31.801456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.802134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.802485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.802764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.877452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:31.879015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.899574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:31.900215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:31.900487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:31.900755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:31.901682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:31.902075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:31.903565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:31.917102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 90/171] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:31.918194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:31.924762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:31.952704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.003040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 82/172] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:32.006841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.048668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4848dc-cbde-4f01-8fd6-a4f4337fdac0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.084199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.085228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.085505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.085770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.087429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=79c132c0cbf74b0899faa3036130f049) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:32.087734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.089477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0130a-6a3f-479b-8638-46cf6c721714 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.095082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 91/173] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:31 2022] GET /identity/v3/role_assignments?user.id=79c132c0cbf74b0899faa3036130f049&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:32.096343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf256c1-0e99-4d47-9a5b-bbd75be12dab None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.096625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf256c1-0e99-4d47-9a5b-bbd75be12dab None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.096895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf256c1-0e99-4d47-9a5b-bbd75be12dab None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.098226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 92/174] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:32.099169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f2874-ca91-479f-9b91-4b1063dab38e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.099451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f2874-ca91-479f-9b91-4b1063dab38e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.099719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f2874-ca91-479f-9b91-4b1063dab38e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.101405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 93/175] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:32.114149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f2874-ca91-479f-9b91-4b1063dab38e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.114893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4aaf25-df86-489b-96d3-f6261fcb4763 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.157426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.158091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.158376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.158649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.159402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:32.159722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.161257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.164639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.168951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 83/176] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=770c965de4374724a8a6de8042997c97 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:32.170932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.239049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.257060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f2874-ca91-479f-9b91-4b1063dab38e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.303824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.304766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.305657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.305933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] PATH_INFO: `/v3/users/7d855e0bceeb417d9bfcb6751dd67251` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.307938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] RBAC: Authorizing `identity:get_user(user_id=7d855e0bceeb417d9bfcb6751dd67251)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.308773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0c0c7d-4894-49f7-8e90-777f121d0dd3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.310627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 94/177] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/users/7d855e0bceeb417d9bfcb6751dd67251 => generated 275 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:32.311782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c05b5e-2445-4e58-96e4-0ab1128edb53 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.312064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c05b5e-2445-4e58-96e4-0ab1128edb53 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.312330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c05b5e-2445-4e58-96e4-0ab1128edb53 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.313723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 95/178] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:32.314755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78454b7-7489-4ccf-9cec-cbcc426464f5 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.315796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78454b7-7489-4ccf-9cec-cbcc426464f5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.316068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78454b7-7489-4ccf-9cec-cbcc426464f5 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.322383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 96/179] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:32.322383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d191834-5a9a-499d-9e21-f2591eff0cd5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.322383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d191834-5a9a-499d-9e21-f2591eff0cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.322383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d191834-5a9a-499d-9e21-f2591eff0cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.332342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fd19-62e7-479e-90ce-1550a22a70c0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.343163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3d191834-5a9a-499d-9e21-f2591eff0cd5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:32.344387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3d191834-5a9a-499d-9e21-f2591eff0cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:32.381945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.382292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.382486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.382759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.384123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.386277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd707df2-4058-468d-905a-d46ee8cf29d6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.388641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 84/180] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:32.389863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.390174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.390449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.412200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:32.413405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:32.434525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 97/181] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:32.435595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5354007-24ad-4ce9-bd03-4dee7a389567 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.435879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5354007-24ad-4ce9-bd03-4dee7a389567 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.436151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5354007-24ad-4ce9-bd03-4dee7a389567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.455063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f5354007-24ad-4ce9-bd03-4dee7a389567 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:32.456241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f5354007-24ad-4ce9-bd03-4dee7a389567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:32.491890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.503767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 85/182] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:32.504810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.534743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 98/183] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:32.536425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.536921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.537430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.556949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:32.558544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:32.574090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.614311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae75d28-4c3c-4c9a-8965-1031640c7eb4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.666031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 99/184] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:32.667089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.673795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.674444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.675188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.775305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.780541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.784188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.813821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086645a3-ed1e-4d3c-b367-376ddbd7288f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:32.870859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:32.871535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:32.871822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:32.872098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:32.872835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:32.873146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:32.874636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:32.890011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 100/185] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=1126c16f80da4dde8de7192a7858171e => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:32.890011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:32.910576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:32.922042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 86/186] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:32.926469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.030422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2e232d-f9d3-40c7-8d8d-b0161a8ff355 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.052183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.056293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d428d208-9018-46d0-89f2-1c058cf131df None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.072674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.073822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.073822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.073822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.085326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:33.091334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:33.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:33.093343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 101/187] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:33.094230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.097771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.098528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.098816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.099916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.112599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:33.117576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:33.118777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:33.119680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:33.120376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 87/188] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:33.120545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.177233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.202005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e0d7a7-2d4c-4f23-abde-4411c31e3fbf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.224189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124f76d-95b4-4183-8347-8e2ae2386250 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.235332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.235959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.236234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.236505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.246256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:33.251506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:33.252547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:33.253308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 102/189] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:33.253925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.262049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.262687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.262967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.263240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] PATH_INFO: `/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.264379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] RBAC: Authorizing `identity:get_project(project_id=0ed5f0d872ce4a048e14eeb0588e0569)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.265829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.267551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 88/190] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/projects/0ed5f0d872ce4a048e14eeb0588e0569 => generated 312 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.268322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.298238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.356831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa31bf8-991a-4cf6-b3cc-f305b5dd6636 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.372655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b6ccb7-6e2c-4052-986a-283374a897a9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.390242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.390862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.391135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.391399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.392537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.393993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.395555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 103/191] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.401313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.403877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.407179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.407787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.408060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.408327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.408988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:33.409287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.410684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.418671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 89/192] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.419447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.502078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8076c9df-7e2b-40d8-9048-93f905f58898 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.503965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.516421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301d129-eb6a-4acf-861a-cf09d6585420 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.534536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.535144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.535420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.535689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.536348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:33.536650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.538035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.546025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 104/193] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.546761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.552626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.553172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.553447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.553716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.554407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:33.554706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.556786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.565043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 90/194] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.565998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.608785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.646146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e19ac-5f7f-49ee-acc5-8ee1ac98367e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.662506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa53e08-6319-446c-b2fe-69c7e654e62d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.678478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.679080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.679362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.679633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.680872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=0ed5f0d872ce4a048e14eeb0588e0569, user.id=7d855e0bceeb417d9bfcb6751dd67251) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:33.681177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.682641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.688262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 105/195] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/role_assignments?user.id=7d855e0bceeb417d9bfcb6751dd67251&scope.project.id=0ed5f0d872ce4a048e14eeb0588e0569&role.id=770c965de4374724a8a6de8042997c97 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.694162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.702235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.702902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.703183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.703459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.704147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.705675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.729346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 91/196] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:33 2022] POST /identity/v3/users => generated 297 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:33.730610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.753641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.833508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb63070-f84b-4463-8318-110d4f8d3a5e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.868457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:33.877843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.878895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.879382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.879721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.899741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.901234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.903009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8124c68d-fd98-409f-9fb0-212507f7f42a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:33.904366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 106/197] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.905361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:33.946956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:33.947160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:33.947404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:33.947673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] PATH_INFO: `/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:33.949911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] RBAC: Authorizing `identity:get_user(user_id=73d6726ac7a8484fbea9e4bc7bc3ac01)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:33.951407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:33.953009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 92/198] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/users/73d6726ac7a8484fbea9e4bc7bc3ac01 => generated 289 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:33.953774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.038641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e16331-c2f6-467e-8550-9e1671126058 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.042278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.082177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e6d222-8f49-4dc6-a38b-094b076c9c01 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.082446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.083099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.083380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.083699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.085566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.087589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.089231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 107/199] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:34.090748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.129167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.129781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.130113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.130384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.153301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.208639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.210546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6480204-15a8-4516-b923-a658ca56b393 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.210823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.257258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.258007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.258803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.259072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.308205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.336988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 93/200] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:34.338265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.359852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.361338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.432911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.459373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c674df-eaed-4d40-8dd6-9f3b1fea778e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.480629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 108/201] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:34.481488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.498855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.500055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.500531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.501006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.548508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.579369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.581350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.582945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad071b-6178-4fb3-b361-92efab2a3762 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.619885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.620469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.621405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.621405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.622293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.623774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.625555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 109/202] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:34.626175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.650406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.697925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 94/203] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:34.703697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.729379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb5e0a0-3960-4f1b-adbc-837868d1bc12 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.764527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.765250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.765475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.765905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.767046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.768593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.770561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 110/204] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:34.771331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.788971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.818710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ac4f3-2b4a-44af-986e-a9ada6b39dde None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.857442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.858456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.858950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.859417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.860447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=097e1916db3d46b1b9fdf67638aea424) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:34.861018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.862690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.866424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:34.870964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 95/205] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/role_assignments?user.id=097e1916db3d46b1b9fdf67638aea424&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=1126c16f80da4dde8de7192a7858171e => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:34.872263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.876752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc7f484-0136-4a57-a997-b69884aeb056 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:34.918172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:34.918172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:34.918500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:34.918745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:34.920018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:34.921568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:34.923638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 111/206] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:34.929618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:34.991889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9e32e9-fc0b-4a19-bac4-a9551972542f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:35.031094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:35.032173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:35.032670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:35.033148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:35.034243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, scope.project.id=027d2665b26c4993a084d20e698ac8eb, user.id=73d6726ac7a8484fbea9e4bc7bc3ac01) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:35.034801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:35.036463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f50ea9-9032-47f6-ab70-d50de735a1f6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:35.037381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fb7bf-8411-419a-bde9-728fddb71c8a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:35.043354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 96/207] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/role_assignments?user.id=73d6726ac7a8484fbea9e4bc7bc3ac01&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:35.076156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:35.076781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:35.077054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:35.077321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:35.078100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=097e1916db3d46b1b9fdf67638aea424) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:35.078407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:35.079992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fecfbb-c931-4161-a0bf-1b69c2d84d32 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:35.086968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 112/208] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:34 2022] GET /identity/v3/role_assignments?user.id=097e1916db3d46b1b9fdf67638aea424&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:36.173744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12138b-2d97-4d6f-96f2-019ffcd776cc None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.174809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12138b-2d97-4d6f-96f2-019ffcd776cc None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.175313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12138b-2d97-4d6f-96f2-019ffcd776cc None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.176966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 97/209] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:36.183141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba06a4f-9625-405a-a4b3-0322785e425e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.184146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba06a4f-9625-405a-a4b3-0322785e425e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.184622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba06a4f-9625-405a-a4b3-0322785e425e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.204860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-1ba06a4f-9625-405a-a4b3-0322785e425e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:36.206492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1ba06a4f-9625-405a-a4b3-0322785e425e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:36.225395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:36.285543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 113/210] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:36.308256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4dae8-0efe-49a0-8f75-a87682eee382 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.308539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4dae8-0efe-49a0-8f75-a87682eee382 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.308805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4dae8-0efe-49a0-8f75-a87682eee382 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.310168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 98/211] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:36.315409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.315697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.315974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.334312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:36.335479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:36.405667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 114/212] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:36.553666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4dae8-0efe-49a0-8f75-a87682eee382 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:36.557415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:36.664394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4dae8-0efe-49a0-8f75-a87682eee382 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:36.671657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:36.700875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:36.701525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.701809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.702113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.714292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:36.719930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:36.720815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:36.722062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 99/213] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:36.729010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:36.754466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:36.773439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cd557a-5253-4e20-95e0-51938f5ad6f3 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:36.799239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:36.809129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:36.809700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.809998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.810270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.811371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:36.812792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:36.818778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 115/214] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:36.819706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:36.845484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e00e8a-65a0-47ff-a9d6-44bcbdd523e5 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:36.884012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:36.885008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.885524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.886130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.887086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:36.887596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:36.889165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:36.904326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 100/215] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:36.906136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:36.921700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe82910-44c3-461b-bdbe-5b067270e7d7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:36.969017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:36.969630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:36.970067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:36.970354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:36.970982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:36.972427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:37.002065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 116/216] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:36 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:47:37.032171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152fc0d4-a916-4130-9223-73f36f139ecc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:37.060040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:37.072680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:37.073476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.073809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.074143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] PATH_INFO: `/v3/users/0b3a3e89630a455bad62f373db6b5c05` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.094898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] RBAC: Authorizing `identity:get_user(user_id=0b3a3e89630a455bad62f373db6b5c05)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:37.096446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc0b1c9-fc4d-4692-813e-e1c68c9d8805 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:37.098603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 101/217] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:36 2022] GET /identity/v3/users/0b3a3e89630a455bad62f373db6b5c05 => generated 297 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:37.104539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:37.117559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:37.222391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662b7d-a181-411b-b22d-a664ffd0b4bb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:37.262036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:37.263001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.263496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.263978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.316803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f2b2a-8ce3-4a2e-a1db-de8f105849b8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.317315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f2b2a-8ce3-4a2e-a1db-de8f105849b8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.317315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f2b2a-8ce3-4a2e-a1db-de8f105849b8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.318877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 102/218] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:37.324058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee9d5-e984-4924-aae0-447b6b2bee6d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.324340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee9d5-e984-4924-aae0-447b6b2bee6d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.324613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee9d5-e984-4924-aae0-447b6b2bee6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.347876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-da2ee9d5-e984-4924-aae0-447b6b2bee6d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:37.350548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-da2ee9d5-e984-4924-aae0-447b6b2bee6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:37.352887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:37.354819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:37.365234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:37.438428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 103/219] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:37.439530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56c01e-8728-4f4f-a9df-e2e41278c8e2 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.439815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56c01e-8728-4f4f-a9df-e2e41278c8e2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.440194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56c01e-8728-4f4f-a9df-e2e41278c8e2 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.441591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 104/220] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:37.448400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.448686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.448960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.472307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:37.473490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:37.480329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 117/221] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:37.487130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:37.567428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 105/222] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:37.608800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca435c43-d43a-4337-a9f6-ed4741ff1d3a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:37.616389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:37.645254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:37.645902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.646207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.646479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.647612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:37.649658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:37.651474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 118/223] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:37 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:37.657404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:37.703155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:37.762874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf57d4f-83f7-4218-8cb1-38916c89c3ad None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:37.764097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:37.816139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:37.817172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.817762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.818371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.819920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=0b3a3e89630a455bad62f373db6b5c05) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:37.821210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:37.822983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:37.837674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 106/224] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:37 2022] GET /identity/v3/role_assignments?user.id=0b3a3e89630a455bad62f373db6b5c05&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=770c965de4374724a8a6de8042997c97 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:37.907682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be018e-7841-4a17-922a-cc6fe024553d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:37.951946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:37.952900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:37.953351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:37.953788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:37.965498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:37.970988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:37.971907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:37.972670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:37.974785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 119/225] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:37.980991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.055901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.110167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c668af-1c1b-4aef-82f9-b03d13a78aad None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.120016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:38.140602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:38.151565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.152189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.152465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.152736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.153408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:38.153708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.155113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.164987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 107/226] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:38.170354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.305305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a956ed01-8407-4010-beaf-d38633d77a00 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.309853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5af7e1b-ab7a-4b16-b280-6d4379d77c8d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.346124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.346762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.347036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.347305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.367851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:38.373173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:38.374030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:38.374797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 120/227] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:38.380930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.381582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.382160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.382457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.383731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.385174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.387084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.388092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 108/228] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:38.393874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.444025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:38.521869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da0c54-9ee9-4bc0-8286-61a98e59dba7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.563776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.564501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.564783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.565053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.566370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.567101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:38.569385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.571648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 109/229] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:38.578969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.584324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cd8f06-0f68-4725-b577-73e28f4753ad None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.658993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.659619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.659904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.660174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.660840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:38.661138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.662539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.683800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 121/230] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:38.701309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef913742-7f7a-4c77-bff3-4adc08c4a109 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.707390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:38.718408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:38.790706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.791389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.791665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.791942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/users/097e1916db3d46b1b9fdf67638aea424/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.793948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, user_id=097e1916db3d46b1b9fdf67638aea424, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.795492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.821569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de01072a-5615-45ae-a8f4-517917f55c71 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:38.832493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 110/231] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:38 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/users/097e1916db3d46b1b9fdf67638aea424/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 10:47:38.868363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:38.869066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:38.869351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:38.869625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:38.870846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:38.872296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:38.873966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 122/232] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:38.882378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:39.053366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a504590-73bc-46d9-82ab-dd751fdbc08b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:39.092003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:39.092651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.092928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.093203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.094388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:39.095858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e4ec70-fdcf-45b6-a7a3-f55e133f0db1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:39.097808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 111/233] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:38 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:39.103371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:39.150961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:39.236417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cdda8d-1950-4467-9c2e-aee9420ce219 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:39.273542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:39.274760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.275039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.275362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/users/097e1916db3d46b1b9fdf67638aea424/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.277570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, user_id=097e1916db3d46b1b9fdf67638aea424, role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:39.279149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c7e82c-a9e1-4fbf-a020-eff25012e136 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:39.294067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 123/234] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:39 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/users/097e1916db3d46b1b9fdf67638aea424/roles/1126c16f80da4dde8de7192a7858171e => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:39.680249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bcd19a-c301-422d-a59f-ab3401cb8181 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.680747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bcd19a-c301-422d-a59f-ab3401cb8181 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.680850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bcd19a-c301-422d-a59f-ab3401cb8181 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.682409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 112/235] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:39.687912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50870-2a5d-4388-8b2e-fd0b597a157c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.688315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50870-2a5d-4388-8b2e-fd0b597a157c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.688483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50870-2a5d-4388-8b2e-fd0b597a157c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.714994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a8a50870-2a5d-4388-8b2e-fd0b597a157c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:39.716198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a8a50870-2a5d-4388-8b2e-fd0b597a157c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:39.811961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 124/236] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:39.906860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5d04-4581-4b44-b673-8145912ad290 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.907447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5d04-4581-4b44-b673-8145912ad290 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.908457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ea5d04-4581-4b44-b673-8145912ad290 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.909439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 113/237] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:39.915603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:39.915893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:39.916246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:39.942399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:39.943509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:39.956975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:40.036219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 125/238] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:40.210209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ea5d04-4581-4b44-b673-8145912ad290 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:40.253916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:40.350730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ea5d04-4581-4b44-b673-8145912ad290 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:40.379251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:40.432245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:40.433197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.434449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.435062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.459127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:40.465258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:40.466420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:40.467389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:40.471544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 114/239] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:40 2022] GET /identity/v3/roles/reader => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:40.477338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:40.536421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f46f-4b94-403c-8a63-fca80c0b6a2c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:40.599361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:40.600009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.600288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.600562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.601706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:40.603999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcdc4a6-3392-4cc2-b1ea-e97b98731e25 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:40.606162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 126/240] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:40.607960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ecd03-41cf-491b-9588-9cf72f0a0390 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.608674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ecd03-41cf-491b-9588-9cf72f0a0390 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.609129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ecd03-41cf-491b-9588-9cf72f0a0390 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.614248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 127/241] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:40 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:40.615293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ecd03-41cf-491b-9588-9cf72f0a0390 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:40.626971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:40.637642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ca1dd6-45e6-42eb-b92b-4ff899fbaf8e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:40.638269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:40.676290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:40.677366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.677876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.678491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.679412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:40.680462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:40.682940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7a7a59-0116-4671-b5ec-ca06030c739f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:40.692651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 115/242] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:40 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:40.694393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.694930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.696080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.717505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:40.719141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:40.751843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ecd03-41cf-491b-9588-9cf72f0a0390 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:40.790216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:40.791237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:40.791718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:40.792239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:40.793121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:40.794886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:40.806057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 116/243] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:40.807255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:40.823700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 128/244] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:40 2022] POST /identity/v3/users => generated 293 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:40.950835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:40.971594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45b3f82-1603-40cc-89f3-12facb52eaa3 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.022846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.023569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.023915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.024190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] PATH_INFO: `/v3/users/9820ee4633134c1ea7509cede440fa83` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.042826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] RBAC: Authorizing `identity:get_user(user_id=9820ee4633134c1ea7509cede440fa83)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.044313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.046095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 117/245] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:40 2022] GET /identity/v3/users/9820ee4633134c1ea7509cede440fa83 => generated 297 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:41.051838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.125533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.161569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.206607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615c14c4-71b8-4c1d-9d5d-72f7c02dc1a7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.247958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.248504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.248781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.249048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.263724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877586fd-2303-4153-b29b-aebf69da4c9b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.306372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.307041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.307403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.307677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.319896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:41.325192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:41.326159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:41.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 118/246] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:41.332886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.336144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.338786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.344405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.423602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.477365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4fce1-634d-4067-a3c4-240042cab1f3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.480373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 129/247] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:41.486583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.517329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.517869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.518170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.518440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.519107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:41.519407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.520791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.533883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 119/248] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:41.534823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.599829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68348af4-98c5-4e65-9e58-d85f89c4fa2a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.606370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.654000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbc68c7-01ba-4b3e-8021-8835f23c3769 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.654512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.655156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.655433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.655701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.657005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.659347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caed6d1d-4293-4da7-8787-dcf0af5dfbe0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.664771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 130/249] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:41.664771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de98a8-5395-4d68-80e7-c7ddddd9afa0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.664771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de98a8-5395-4d68-80e7-c7ddddd9afa0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.664771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de98a8-5395-4d68-80e7-c7ddddd9afa0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.665242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 131/250] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:41.677856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.678262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.678662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.706968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:41.708705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:41.709754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.710516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.710941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.711336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] PATH_INFO: `/v3/users/0b3a3e89630a455bad62f373db6b5c05` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.712541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] RBAC: Authorizing `identity:get_user(user_id=0b3a3e89630a455bad62f373db6b5c05)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.716149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.716149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 120/251] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/users/0b3a3e89630a455bad62f373db6b5c05 => generated 297 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:41.716434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.748525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.797295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 132/252] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:41.800871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:41.868030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c3a7d-c310-49f4-bb75-a963c03622f7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:41.939718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:41.948597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:41.949223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:41.949500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:41.949771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:41.950538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=9820ee4633134c1ea7509cede440fa83) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:41.950843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:41.953578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:41.968215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 121/253] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/role_assignments?user.id=9820ee4633134c1ea7509cede440fa83&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 259 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:42.008684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adbe524-3a51-4203-b08b-c32dccca5aa9 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:42.057118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:42.057684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.057961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.058252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.062852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:42.064343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d775d-f977-47e4-9048-31f22d397e8b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:42.066319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 133/254] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:41 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:42.073222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:42.110534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0fb1a2-1930-4887-9b34-b1f855656dbc None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.110839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0fb1a2-1930-4887-9b34-b1f855656dbc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.111084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0fb1a2-1930-4887-9b34-b1f855656dbc None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.112650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 134/255] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:42.118319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.118627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.118884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.159368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:42.160622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:42.212895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:42.274173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 135/256] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:42.275052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:42.282963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fbb5b-c76c-47d7-b8fc-f0b3847ceb81 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:42.356584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:42.357279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.357555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.357826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/users/0b3a3e89630a455bad62f373db6b5c05/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.359790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, user_id=0b3a3e89630a455bad62f373db6b5c05, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:42.361324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:42.395812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 122/257] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:42 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/users/0b3a3e89630a455bad62f373db6b5c05/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:42.417109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8db9a-b41f-4276-9d51-4409ba1dc5b6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:42.462232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:42.462914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.463185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.463461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.484220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:42.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:42.490891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:42.491692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:42.494274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 136/258] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:42.502012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:42.594410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:42.651678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0088a4c-a313-4317-b616-9778b009c745 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:42.687771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:42.688408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.688683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.688958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.689653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:42.689963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:42.691435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:42.701772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 123/259] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:42.707016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:42.749234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:42.818905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:42.838011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095581f-71ee-476d-865f-c1ae48e6c1b1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:42.877949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:42.880247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:42.880916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:42.881995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:42.884603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:42.886450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:42.893539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 137/260] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:42 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:42.901543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.028195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000364dd-7607-402e-9452-e70c97548c0c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.057583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.066296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.069989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe69a30-fdbe-4e52-916a-69aced5badfa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.099791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.100647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.101020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.101386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:43.124818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:43.130841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:43.131694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:43.132915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 124/261] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:43.139728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.196083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.196710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.196993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.197270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:43.339610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:43.341097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:43.363888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855b748e-566b-44ae-981b-fa71e282a215 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.419716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.420331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.420604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.420872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:43.421537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:43.421835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:43.423440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:43.448192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 125/262] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:43.465264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.485506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.502535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 138/263] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 601 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:43.508583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.550333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.656284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392f8234-d33e-4927-8ea7-ce74b395fddd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.721922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3569d8-1b2d-4449-8189-b04686df54d6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.727633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.728289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.728569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.729099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] PATH_INFO: `/v3/users/097e1916db3d46b1b9fdf67638aea424` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:43.730394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] RBAC: Authorizing `identity:get_user(user_id=097e1916db3d46b1b9fdf67638aea424)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:43.731851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:43.733567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 126/264] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:43 2022] GET /identity/v3/users/097e1916db3d46b1b9fdf67638aea424 => generated 283 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:43.739226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.798080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.799799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.800573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.801339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:43.806814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:43.808575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:43.810522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 139/265] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:43 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:43.816225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:43.894249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcfc1dc-0286-4b43-a939-23f43d0e9f63 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:43.905325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.931047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:43.941885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:43.942628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:43.942970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:43.943307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.003303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec6766c-4779-4408-b46b-a08074e827b8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:44.023372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:44.025197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:44.072697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:44.073725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.074210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.075590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=097e1916db3d46b1b9fdf67638aea424) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:44.076074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:44.077898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:44.090896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 140/266] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:43 2022] GET /identity/v3/role_assignments?user.id=097e1916db3d46b1b9fdf67638aea424&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:44.147535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 127/267] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 10:47:44.152774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:44.306424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976683e-3fec-43a7-8717-979f271a60cb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:44.341800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:44.356389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:44.357041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.357318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.357595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.362930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:44.364443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527a6160-069b-40c7-a112-fa8454780305 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:44.366620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 141/268] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:44 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:44.373058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:44.390583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:44.546494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6878cfa-ddfc-441a-9960-92f5c82453e3 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.547147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6878cfa-ddfc-441a-9960-92f5c82453e3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.547636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6878cfa-ddfc-441a-9960-92f5c82453e3 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.549042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8743fc1-312c-4b31-a2e3-01bb48640f9d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:44.549735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 142/269] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:44.557074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a71ce3-5732-48f8-88c6-014bb66d3a1f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.557640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a71ce3-5732-48f8-88c6-014bb66d3a1f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.558132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a71ce3-5732-48f8-88c6-014bb66d3a1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.580548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-71a71ce3-5732-48f8-88c6-014bb66d3a1f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:44.582098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-71a71ce3-5732-48f8-88c6-014bb66d3a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:44.610681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:44.611294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.611569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.611838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.612563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=097e1916db3d46b1b9fdf67638aea424) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:44.612868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:44.614338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bbe678-7b36-47fd-819a-96644c9c6d89 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:44.627799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 128/270] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:44 2022] GET /identity/v3/role_assignments?user.id=097e1916db3d46b1b9fdf67638aea424&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=1126c16f80da4dde8de7192a7858171e => generated 608 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:44.661875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 143/271] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:44.939548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afe4349-9105-401c-91cb-1576a0f7f2f9 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.940098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afe4349-9105-401c-91cb-1576a0f7f2f9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.940098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afe4349-9105-401c-91cb-1576a0f7f2f9 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.941427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 129/272] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:44.947086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44b064-33d0-4618-9415-78325882f0b8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:44.947964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44b064-33d0-4618-9415-78325882f0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:44.948430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44b064-33d0-4618-9415-78325882f0b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:44.960157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afe4349-9105-401c-91cb-1576a0f7f2f9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:44.975482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cb44b064-33d0-4618-9415-78325882f0b8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:44.977682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cb44b064-33d0-4618-9415-78325882f0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:45.074385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 144/273] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:45.093411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.127956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afe4349-9105-401c-91cb-1576a0f7f2f9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.164844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da6f42d-d09a-479d-97cf-2037c4d36a85 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.165219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da6f42d-d09a-479d-97cf-2037c4d36a85 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.165507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da6f42d-d09a-479d-97cf-2037c4d36a85 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.166814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.167216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 145/274] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:47:45.167911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.168453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.168911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.172940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.173223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.173511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.187858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:45.193890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:45.194752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:45.196084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:45.197947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 130/275] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:45.202823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:45.204366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:45.205359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.274850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.303070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 146/276] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:45.334876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758ddc5a-2b51-4aa2-bab6-47b7ef34d46d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.368097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.373803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.374844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.375468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.376105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.376769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:45.377117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:45.378584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:45.386359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.387158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 131/277] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:45.392731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.497623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddde559-a2c4-4f55-b71e-a5b8209a36b3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.497795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f96c6b-db6c-46fd-96d8-86c5d225aafa None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.532821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.533087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.533547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.533727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.533819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.534021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.534116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] PATH_INFO: `/v3/users/95d14ce1d205411a9cd57681dcdf7055` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.534290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.535379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:45.536833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683553df-5550-4da3-b9ca-2be3303ba736 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:45.538543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 147/278] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:45.539702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfd9f4-ee07-4809-82f4-7c678962656c None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.539985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfd9f4-ee07-4809-82f4-7c678962656c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.540259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cfd9f4-ee07-4809-82f4-7c678962656c None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.545666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 148/279] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:45.546506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cfd9f4-ee07-4809-82f4-7c678962656c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.549809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] RBAC: Authorizing `identity:get_user(user_id=95d14ce1d205411a9cd57681dcdf7055)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:45.551519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05934e-5309-45dd-9ac9-edb1a8817213 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:45.553610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 132/280] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/users/95d14ce1d205411a9cd57681dcdf7055 => generated 293 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:45.555719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.555995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.556263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.575622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:45.576918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:45.601184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.623891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.644490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 133/281] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:45.645308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.654251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cfd9f4-ee07-4809-82f4-7c678962656c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.692729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.693664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.694230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.694646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.696011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:45.697757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:45.723324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 149/282] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:47:45 2022] POST /identity/v3/users => generated 293 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:45.724704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.754194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3898ed-9453-48ed-9dfc-4c2ab59c3c63 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.799382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.800401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.800919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.801458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.802556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, user.id=95d14ce1d205411a9cd57681dcdf7055, scope.system=all) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:45.803072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:45.805350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:45.811647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 134/283] 10.222.0.46 () {56 vars in 1308 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/role_assignments?user.id=95d14ce1d205411a9cd57681dcdf7055&scope.system=all&role.id=770c965de4374724a8a6de8042997c97 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:45.856898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c5e69b-3777-4430-8dc0-ce5daf86131b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:45.865134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:45.900195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:45.901004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:45.901290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:45.901608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:45.914915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:45.920776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:45.922188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:45.923678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 150/284] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:45.924307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.929687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:45.983162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.049856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443fdadb-6aa7-4ba1-8550-9ec3a8c6b6fc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.053710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fad31-2af8-4876-a4c3-39dacf542633 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.094186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.095188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.095669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.096142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.097053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.097678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.097957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.098277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.098951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:46.099286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.101191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.111571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:46.113960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 151/285] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:46.117968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:46.118929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:46.119939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:46.121590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.122391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 135/286] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:46.130101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.158249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.255160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c47ae-74e7-4b58-a319-364699054f11 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.259252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d528d-09dc-4807-b300-55465a6d3d2a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.267006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.300452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.301222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.301598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.301877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] PATH_INFO: `/v3/users/9820ee4633134c1ea7509cede440fa83` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.302622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.303147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] RBAC: Authorizing `identity:get_user(user_id=9820ee4633134c1ea7509cede440fa83)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.303266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.303497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.303765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.304434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:46.304628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.304790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.306210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.306574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 152/287] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/users/9820ee4633134c1ea7509cede440fa83 => generated 297 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:46.315599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.318131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 136/288] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:46.332298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.482621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.488309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94c3c54-2594-4a09-a196-689d6eaec7cf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.531499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f06799-1633-46e6-acd4-814df4642dca None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.553967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.555006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.555500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.555980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.557432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.559114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.565685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 153/289] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:46.574973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.587662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.588313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.588593 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.588867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] PATH_INFO: `/v3/users/0b3a3e89630a455bad62f373db6b5c05` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.590055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] RBAC: Authorizing `identity:get_user(user_id=0b3a3e89630a455bad62f373db6b5c05)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.591518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.593579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 137/290] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/users/0b3a3e89630a455bad62f373db6b5c05 => generated 297 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:46.601679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.666389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.728700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6fb53-d8a2-41cc-b4d9-37d486aee855 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.766971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.767615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.768121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.768391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.845743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.847279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.849789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6609c23-62ec-420a-b9a2-5add38860208 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:46.894464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:46.942519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:46.943198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:46.943481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:46.943761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/users/9820ee4633134c1ea7509cede440fa83/roles/a6d2f4a8b54a44b29b129335ac106b7b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:46.946302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, user_id=9820ee4633134c1ea7509cede440fa83, role_id=a6d2f4a8b54a44b29b129335ac106b7b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:46.947755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:46.962791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 138/291] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 10:47:46.969698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:46.981510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 154/292] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:47:46 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/users/9820ee4633134c1ea7509cede440fa83/roles/a6d2f4a8b54a44b29b129335ac106b7b => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:47.122011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebc0b29-1d95-41b3-8d0e-2da6442274cd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:47.172217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:47.172866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:47.173149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:47.173425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:47.174608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:47.176051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ad6265-1c0b-4f61-8855-0e2ed30f806f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:47.178035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 139/293] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:46 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:47.186183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:47.215614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:47.342813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dd0cc2-5051-4928-8ccd-978360e55488 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:47.393684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:47.394240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:47.394521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:47.394792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:47.395521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=0b3a3e89630a455bad62f373db6b5c05) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:47.395831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:47.397272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd7402c-1bf2-44be-b3f2-26ab363e5a6e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:47.406614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 155/294] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:47 2022] GET /identity/v3/role_assignments?user.id=0b3a3e89630a455bad62f373db6b5c05&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=770c965de4374724a8a6de8042997c97 => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:48.204894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fe0235-ffa5-4ea2-aaf1-003bb7257dc8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.205405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fe0235-ffa5-4ea2-aaf1-003bb7257dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.205405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fe0235-ffa5-4ea2-aaf1-003bb7257dc8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.207085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 140/295] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:48.212414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb155587-84b8-4cad-ad61-5f5d0b1a3228 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.212848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb155587-84b8-4cad-ad61-5f5d0b1a3228 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.213028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb155587-84b8-4cad-ad61-5f5d0b1a3228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.232288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-bb155587-84b8-4cad-ad61-5f5d0b1a3228 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:48.233476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bb155587-84b8-4cad-ad61-5f5d0b1a3228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:48.310150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 156/296] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:48.486067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dfd8e6-900d-4a51-8dd3-938adc4968be None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.486381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dfd8e6-900d-4a51-8dd3-938adc4968be None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.486567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dfd8e6-900d-4a51-8dd3-938adc4968be None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.487914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 141/297] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:48.494534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f0f53e-4919-4644-80dc-65d463840ab8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.494822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f0f53e-4919-4644-80dc-65d463840ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.495439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f0f53e-4919-4644-80dc-65d463840ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.514507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-14f0f53e-4919-4644-80dc-65d463840ab8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:48.515692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-14f0f53e-4919-4644-80dc-65d463840ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:48.529828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:48.590116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 157/298] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:48.596225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec7e4d-c03a-4cb0-84ff-c048bc4f7883 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.596731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec7e4d-c03a-4cb0-84ff-c048bc4f7883 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.597235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec7e4d-c03a-4cb0-84ff-c048bc4f7883 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.598790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 142/299] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:48.604425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.604943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.605419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.606038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ec7e4d-c03a-4cb0-84ff-c048bc4f7883 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:48.610910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:48.630100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:48.632557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:48.726179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 158/300] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:48.740014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ec7e4d-c03a-4cb0-84ff-c048bc4f7883 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:48.776162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:48.776802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.777080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.777354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.788270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:48.794040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:48.794898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:48.795685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 143/301] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:48.801159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:48.860851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:48.888087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:48.917416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ee3c39-9f6d-4fca-b94c-ac9609d3e594 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:48.946169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:48.959562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca564078-253b-468c-8026-748549b2d066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:48.960312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca564078-253b-468c-8026-748549b2d066 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:48.960708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca564078-253b-468c-8026-748549b2d066 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:48.961616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca564078-253b-468c-8026-748549b2d066 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:48.962439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca564078-253b-468c-8026-748549b2d066 None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:48.962839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca564078-253b-468c-8026-748549b2d066 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:48.964434 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca564078-253b-468c-8026-748549b2d066 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:48.978070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 159/302] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:48.979976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca564078-253b-468c-8026-748549b2d066 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:48.994280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722f8d9c-b50b-41fb-95c1-4eab8ba99312 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.033781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.034456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.034737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.035013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.036149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.037555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.039837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 144/303] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:49.040611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.087390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca564078-253b-468c-8026-748549b2d066 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.124333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.124974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.125257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.125538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] PATH_INFO: `/v3/users/95d14ce1d205411a9cd57681dcdf7055` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.126636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] RBAC: Authorizing `identity:get_user(user_id=95d14ce1d205411a9cd57681dcdf7055)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.128625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.130248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 160/304] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:48 2022] GET /identity/v3/users/95d14ce1d205411a9cd57681dcdf7055 => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:49.131080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.145033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:49.153468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576c59c1-57c5-421f-9f77-29b211dc6e1d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.185020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:49.190782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.191419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.191695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.191965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.203443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:49.208179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:49.209538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 145/305] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:49.211329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.251659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434823b8-d561-4db9-947d-9763fac9aacc None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.293655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.294327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.294612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.294885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] PATH_INFO: `/v3/groups` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.295543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.297035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.313873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 161/306] 10.222.0.46 () {60 vars in 1134 bytes} [Fri Oct 28 10:47:49 2022] POST /identity/v3/groups => generated 229 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:47:49.319679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.338803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c283aae-85a6-43ca-bea7-3db6e085d6a6 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.381043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.382070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.382596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.383073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] PATH_INFO: `/v3/system/users/95d14ce1d205411a9cd57681dcdf7055/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.384975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=95d14ce1d205411a9cd57681dcdf7055, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.386640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a7e389-c6ef-447d-a48d-0701c3b64379 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.399487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 146/307] 10.222.0.46 () {58 vars in 1334 bytes} [Fri Oct 28 10:47:49 2022] PUT /identity/v3/system/users/95d14ce1d205411a9cd57681dcdf7055/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:49.401274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a09ce04-7f78-453f-ab88-92a1cd0022d1 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.401772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a09ce04-7f78-453f-ab88-92a1cd0022d1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.402626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a09ce04-7f78-453f-ab88-92a1cd0022d1 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.404639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 147/308] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:49.415472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.416055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:49.416349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.416647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.444012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:49.445928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:49.466177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471d9fbf-8502-4065-bfda-915a30e77448 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.515912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.516463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.516740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.517010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.517696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:49.518028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.519627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.523392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:49.530976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 162/309] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:49 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:49.536790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.545934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 148/310] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:49.654551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847bf6d6-d7ca-47e4-b563-a933770dd685 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.684473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:49.690350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.691008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.691296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.695703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] PATH_INFO: `/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.705885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] RBAC: Authorizing `identity:get_user(user_id=1d8ce75cc349406fa01e3dd8f7ab2603)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.707688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.712944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 163/311] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:49 2022] GET /identity/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:49.716499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.820723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:49.828685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c81297-ab88-4bed-a785-ac68bd7161a5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.865647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c759335c-6891-4498-94ee-19394b041319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.866632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c759335c-6891-4498-94ee-19394b041319 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.867162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c759335c-6891-4498-94ee-19394b041319 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.867652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c759335c-6891-4498-94ee-19394b041319 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:49.868565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c759335c-6891-4498-94ee-19394b041319 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, user.id=1d8ce75cc349406fa01e3dd8f7ab2603, scope.system=all) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:49.869136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c759335c-6891-4498-94ee-19394b041319 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:49.870812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c759335c-6891-4498-94ee-19394b041319 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:49.876271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 149/312] 10.222.0.46 () {56 vars in 1308 bytes} [Fri Oct 28 10:47:49 2022] GET /identity/v3/role_assignments?user.id=1d8ce75cc349406fa01e3dd8f7ab2603&scope.system=all&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:49.951103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e239e47-25c1-49c1-afec-7a3f6e6020d2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:49.990656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:49.991309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:49.991589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:49.992436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.004504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:50.010464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:50.011393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:50.012158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 164/313] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:50.018982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c759335c-6891-4498-94ee-19394b041319 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:50.074961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:50.128094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c759335c-6891-4498-94ee-19394b041319 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:50.164255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:50.165257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:50.165729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:50.166244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.167112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:50.167615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:50.169203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:50.178326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 150/314] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:50.184808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:50.244098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:50.305676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44de5413-b8fe-4e59-b766-b3d05a621095 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:50.342699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:50.343261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:50.343600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:50.343880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] PATH_INFO: `/v3/users/9820ee4633134c1ea7509cede440fa83` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.345026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] RBAC: Authorizing `identity:get_user(user_id=9820ee4633134c1ea7509cede440fa83)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:50.346497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:50.348444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 165/315] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:50 2022] GET /identity/v3/users/9820ee4633134c1ea7509cede440fa83 => generated 297 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:50.354526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:50.460097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6c9229-defc-4e94-9e97-a0624a0981f1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:50.473652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:50.497040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:50.497645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:50.497919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:50.498217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.572197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:50.573691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:50.677886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 151/316] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:50.687071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:50.796024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d7d27-6c42-4b4e-b6b4-cb2b99ff4110 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:50.814866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:50.836418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:50.837058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:50.837333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:50.837606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.838768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:50.840275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5a6fa1-591a-46bc-8da5-e24b4684ce08 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:50.841903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 166/317] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:50 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:50.847984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:50.862969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:50.955487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ba24a-ccaa-45eb-b40b-959eff169800 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:50.990292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:50.990951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:50.991230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:50.991501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:50.992794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, scope.project.id=519f588961584ff99d94b1382275b6e9, user.id=9820ee4633134c1ea7509cede440fa83) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:50.993099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:50.994561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e9a443-a449-43b8-96e5-fbfefc9236f9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:51.000395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 152/318] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:47:50 2022] GET /identity/v3/role_assignments?user.id=9820ee4633134c1ea7509cede440fa83&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:51.593261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc28f6-6734-4c8d-86f4-e26dc271f876 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:51.593763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc28f6-6734-4c8d-86f4-e26dc271f876 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:51.593837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc28f6-6734-4c8d-86f4-e26dc271f876 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:51.595303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 167/319] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:51.600314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c76b8-49c5-4345-8e8e-b855bb67a0fb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:51.600689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c76b8-49c5-4345-8e8e-b855bb67a0fb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:51.600851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c76b8-49c5-4345-8e8e-b855bb67a0fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:51.620806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d63c76b8-49c5-4345-8e8e-b855bb67a0fb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:51.622068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d63c76b8-49c5-4345-8e8e-b855bb67a0fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:51.691138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 153/320] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:51.731706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f1963-5a36-4ae4-8a19-d146bc2fd235 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:51.732000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f1963-5a36-4ae4-8a19-d146bc2fd235 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:51.732265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f1963-5a36-4ae4-8a19-d146bc2fd235 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:51.733712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 168/321] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:51.738766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:51.739059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:51.739339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:51.752106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:51.759520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:51.760725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:51.827324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 154/322] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:51.963146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f1963-5a36-4ae4-8a19-d146bc2fd235 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.067924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f1963-5a36-4ae4-8a19-d146bc2fd235 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.094284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.104362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.105404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.105684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.105954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.107697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:52.109145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:52.110925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 169/323] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:47:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:52.116057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.171337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:52.197008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3cda1f-6e1b-4a63-9b32-4c00f633b56d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.218688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8089cc-0199-4aca-b072-c0f14c561088 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.230627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.231258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.231538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.231812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.234413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:52.243406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-b648a5ea-2651-4d75-966c-da31e0cdd2e0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:52.248343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:52.249225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:52.250395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 155/324] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:52.251069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f62742-94c1-4467-acd6-20ae65625679 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.251069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f62742-94c1-4467-acd6-20ae65625679 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.251391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f62742-94c1-4467-acd6-20ae65625679 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.252944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 156/325] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:52.255331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f62742-94c1-4467-acd6-20ae65625679 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.258720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.259336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.259611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.259885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] PATH_INFO: `/v3/groups` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.260520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:52.261941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df13561-39f9-47dd-9b9b-f25f82ca7b4a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:52.277464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 170/326] 10.222.0.46 () {60 vars in 1134 bytes} [Fri Oct 28 10:47:52 2022] POST /identity/v3/groups => generated 226 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:47:52.278524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.278804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.279073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.297938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:52.299253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:52.377236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 171/327] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:52.387941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f62742-94c1-4467-acd6-20ae65625679 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.425608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.426268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.426545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.426816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.427495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:52.427795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:52.429189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:52.439200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 157/328] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:47:52.447117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.497144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:52.561818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa3f758-9659-4bc8-9aa7-1d87b8159090 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.599055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.599801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.599982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.600250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] PATH_INFO: `/v3/users/95d14ce1d205411a9cd57681dcdf7055` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.601460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] RBAC: Authorizing `identity:get_user(user_id=95d14ce1d205411a9cd57681dcdf7055)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:52.602940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:52.605261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 172/329] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/users/95d14ce1d205411a9cd57681dcdf7055 => generated 293 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:52.611801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.664720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.684608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:52.735315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306691ce-a004-4c29-9a6a-e0ad1d4028ea None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.781959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.782631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.783210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.783210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.783893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] RBAC: Adding query filter params (role.id=770c965de4374724a8a6de8042997c97, user.id=95d14ce1d205411a9cd57681dcdf7055, scope.system=all) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:52.784202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:52.785640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:52.796597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff07675-ff92-4550-b8ca-21bcca1dd63f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:52.799127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 158/330] 10.222.0.46 () {56 vars in 1308 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/role_assignments?user.id=95d14ce1d205411a9cd57681dcdf7055&scope.system=all&role.id=770c965de4374724a8a6de8042997c97 => generated 510 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:52.813652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:52.845575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:52.846277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:52.846559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:52.846832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:52.865309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:52.872716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:52.873662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:52.874453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 173/331] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:52.880320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:52.998154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c254873-c6cf-4e2a-bc84-dd05936cd838 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:53.016961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:53.036505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:53.037099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:53.037381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:53.037706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:53.038469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:53.039324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:53.040748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:53.050265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 159/332] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:53.056401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:53.175754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439afc6c-8e22-4857-ad76-091fbce33f0e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:53.219369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:53.220018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:53.220300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:53.220574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] PATH_INFO: `/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:53.221733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] RBAC: Authorizing `identity:get_user(user_id=1d8ce75cc349406fa01e3dd8f7ab2603)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:53.223243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a615342c-e900-4cd3-93f1-ff7cb50cb584 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:53.225124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 174/333] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:53 2022] GET /identity/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603 => generated 293 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:53.231560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:53.344105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80eb569d-2448-4ae1-9bd5-5f221a1efeb7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:53.383775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:53.384733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:53.385161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:53.385625 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] PATH_INFO: `/v3/system/users/1d8ce75cc349406fa01e3dd8f7ab2603/roles/a6d2f4a8b54a44b29b129335ac106b7b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:53.387253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1d8ce75cc349406fa01e3dd8f7ab2603, role_id=a6d2f4a8b54a44b29b129335ac106b7b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:53.388791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d2ee21-ae1a-4e1b-9568-a1ff5ef09866 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:53.400618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 160/334] 10.222.0.46 () {58 vars in 1334 bytes} [Fri Oct 28 10:47:53 2022] PUT /identity/v3/system/users/1d8ce75cc349406fa01e3dd8f7ab2603/roles/a6d2f4a8b54a44b29b129335ac106b7b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:47:55.373423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38a14b-f9c6-41c0-9902-26d5384466d9 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.373917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38a14b-f9c6-41c0-9902-26d5384466d9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.374046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38a14b-f9c6-41c0-9902-26d5384466d9 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.375695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 175/335] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:55.381058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1f135-4f86-465b-9348-e076b48fbae8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.381472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1f135-4f86-465b-9348-e076b48fbae8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.381659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a1f135-4f86-465b-9348-e076b48fbae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.391473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab97f51-044c-45a5-8816-6973421d12cf None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.391760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab97f51-044c-45a5-8816-6973421d12cf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.392036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab97f51-044c-45a5-8816-6973421d12cf None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.393885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 176/336] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:55.398912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.399195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.399470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.410939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a1a1f135-4f86-465b-9348-e076b48fbae8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:55.412143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a1a1f135-4f86-465b-9348-e076b48fbae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:55.431344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:55.432849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:55.470443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:55.486551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:55.534608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 161/337] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:55.535961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fce3f7-3bac-46eb-b3d7-9f6da125afd6 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.536252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fce3f7-3bac-46eb-b3d7-9f6da125afd6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.536528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fce3f7-3bac-46eb-b3d7-9f6da125afd6 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.537762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 162/338] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:55.544606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:55.544888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:55.545161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:55.574867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 177/339] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:47:55.579066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:55.580262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:55.677347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 163/340] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:56.069739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.093998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.283195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fe269e-6f0d-494c-92b4-f39f7825faa7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.301932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5901aecd-c6ee-4473-8e7b-3b68cd8cedb6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.306262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.330087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.330742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.331018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.331290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.356917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:56.361808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:56.362695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:56.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 178/341] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:56.369273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.374318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.374970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.375249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.375523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.388512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:56.395217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:56.396083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:56.396848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:56.399107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 164/342] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:47:56.400152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.401827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.488982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c69d0-d46b-40a8-8727-c526bc7b9f66 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.532533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.533564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.534108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.534573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.546682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d8d1002e-1910-4ec3-b0a1-9335316f9aa9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:56.552237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:56.553134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:47:56.553916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:56.556917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 179/343] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:56.558454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f20af4-0b65-4de7-83f4-8cb93ea5577d None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.558944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f20af4-0b65-4de7-83f4-8cb93ea5577d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.559503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f20af4-0b65-4de7-83f4-8cb93ea5577d None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.561478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 180/344] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:56.564273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f20af4-0b65-4de7-83f4-8cb93ea5577d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.590454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8691a6d6-82d6-4f18-bf2d-77003a19dcd1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.631486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.632867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.633304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.633576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.636126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:56.636126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:56.636126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:56.646430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 165/345] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:56.646831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.685395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.688702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.705820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f20af4-0b65-4de7-83f4-8cb93ea5577d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.744677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.745292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.745569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.745840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.746536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:56.746837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:56.748206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768f1517-b68e-4fee-b086-564e33c5f0bf None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:56.757390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 181/346] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:56.759159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.759440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.759712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.780546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:56.781468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb28ade-db0a-443a-ae02-baa4342456cb None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.781729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:56.829745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:56.830394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.830674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.830943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.831611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:56.831911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:56.833696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988ac534-7f2e-499c-98e1-b149bf436196 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:56.847342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 166/347] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:56.848502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6af0f0-f1d4-4d44-8e37-fe5b6874b8c6 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:56.848785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6af0f0-f1d4-4d44-8e37-fe5b6874b8c6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:56.849055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6af0f0-f1d4-4d44-8e37-fe5b6874b8c6 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:56.855208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 167/348] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:56.855208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6af0f0-f1d4-4d44-8e37-fe5b6874b8c6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.870376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 182/349] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:56.871476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:56.967377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.974411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:56.988510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6af0f0-f1d4-4d44-8e37-fe5b6874b8c6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:56.989333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f1f41-0c53-4d22-bbbd-1920faf108be None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.031364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.031978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.032261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.032535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.035485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.036325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.036896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.037359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.111722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.113108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fc7402-065e-4352-85f2-1f77f97bb79c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.118604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.120342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.217675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.220149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.242063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 168/350] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:57.242655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 183/351] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:57.243456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.243565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.243739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.244010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.271328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:57.271510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:57.367220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 169/352] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:57.369083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.378874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594d5c1-cc7f-4029-bbbd-dcb87caacae4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.424154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.424910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.425115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.425398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.492823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.506928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f1df1-7fcc-43bc-9bc4-2c4a5a4405c1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.516938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.518505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.525299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.549259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.549898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.550213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.550483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.563650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:57.568607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:57.569475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:47:57.570880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:57.572002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 170/353] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:57.572002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.661590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 184/354] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:57.662392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.692234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39301-e839-4d33-9cac-463457410938 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.735706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.736336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.736610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.738015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.738015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.739495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce04c43-7cf8-45ab-9162-bd51301cdc9c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.741107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 171/355] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:57.746898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e298-32bf-4df0-b889-c7f5775a453c None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.746898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e298-32bf-4df0-b889-c7f5775a453c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.746898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475e298-32bf-4df0-b889-c7f5775a453c None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.746898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 172/356] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:47:57.746898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475e298-32bf-4df0-b889-c7f5775a453c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.766545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.792155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099430d7-185b-4906-a281-7db9d4337e1e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.796729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:57.834167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.834824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.835659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.835659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.836979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.839151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.841576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 185/357] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:57.842578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.867901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475e298-32bf-4df0-b889-c7f5775a453c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:57.906619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:57.907470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:57.907753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:57.908028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:57.908697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:57.909000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:57.910514 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:57.919954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 173/358] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:57.922878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:57.969548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f884078-acff-43bc-becc-59f53debc1ac None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.020319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.020963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.021320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.021522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.024213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.025043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.026730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 186/359] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.027496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.048219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.057747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33065136-7225-4543-bc49-35bd2bed6c2d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.064649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.098108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.098807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.099867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.099867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.112076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fa143e3f-24be-4cb6-a7b0-42a9fef253c9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:58.118720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:58.119578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:47:58.120344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 174/360] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:58.121267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.121828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.121828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.141085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:47:58.142360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:47:58.175118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bf0de2-8a76-4747-863b-cd305b58af7c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.217767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 175/361] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:47:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:47:58.218566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.228656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.229561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.229838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.230217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.236537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.238191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.240500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 187/362] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.241273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.349048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.360588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707d2542-10e5-415f-b808-b36a5b628095 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.362238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.406699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1c0b6-b7f7-4fda-b7ca-627532c18ff5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.407334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.408195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.408649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.409093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.410647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.412463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.414954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 176/363] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.415726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.458746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.459386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.459557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.459834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.542645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.543204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8eaabf-f6aa-406e-97a6-8647f1c32c6f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.544256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.590713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.591369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.591644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.591913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.593044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.594521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.596088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 177/364] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.597207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.629106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.637351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.658171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 188/365] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:47:58.659009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.706461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512afaec-fb75-4aa7-851c-34ee8cf216f5 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.740163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.740697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.740976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.741249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.741949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:58.742277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.743648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.751774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 178/366] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.752468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.760227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed46b6-3916-499a-9fd8-0a15ac76d580 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.795332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.795947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.796228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.796817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.797552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=770c965de4374724a8a6de8042997c97, scope.project.id=519f588961584ff99d94b1382275b6e9) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:58.797859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.799334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.804649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 189/367] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=770c965de4374724a8a6de8042997c97 => generated 260 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.805417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.858785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.864462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:58.873657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d871f9b-5cff-4111-817e-a02b4677ce9f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.910463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.911087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.911367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.911638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.912375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=027d2665b26c4993a084d20e698ac8eb) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:58.912684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:58.914415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:58.920196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 179/368] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=1126c16f80da4dde8de7192a7858171e => generated 260 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:58.923249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:58.924553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1662e9b-8d07-49c8-82a3-83bea0d4a655 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:58.969639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:58.970548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:58.970833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:58.971182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:58.986172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:47:58.991393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:47:58.992876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:47:58.993746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:47:58.994370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 190/369] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:47:58.994698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.089382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b46f5cb-4b8a-4395-ac27-96ba75d4b796 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.127339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.128221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.128715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.129186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.130220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=770c965de4374724a8a6de8042997c97, scope.project.id=027d2665b26c4993a084d20e698ac8eb) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:59.130734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.133041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.138830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 180/370] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=770c965de4374724a8a6de8042997c97 => generated 260 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.139868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.146847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856e1b7-0d26-42f0-ba5a-7da75c561ac2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.150356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:59.170346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:59.200227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.200925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.201201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.201479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.210973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.214556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.218731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 191/371] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:47:58 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.226574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.285130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4573164e-98f1-47f9-8a1d-10297748a5b9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.326459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.327189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.327478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.327757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] PATH_INFO: `/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.328972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] RBAC: Authorizing `identity:get_user(user_id=1d8ce75cc349406fa01e3dd8f7ab2603)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.330848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.332587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 181/372] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/users/1d8ce75cc349406fa01e3dd8f7ab2603 => generated 293 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.333831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.483820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aec4afd-38e7-43e8-9ad3-65ccd7f5f77c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.499433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:59.520119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94e4ab-0164-4be6-9aab-c504f61ea3c4 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.532497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.533056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.533339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.533613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.534312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:59.534616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.535981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.552574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 192/373] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.553414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.592704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.593422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.593629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.593904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.598487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.604334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.607374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 182/374] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.608001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.629920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:59.705130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3540d-20db-4f5b-a2ba-0bca1a36cea6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.745173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.745883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.746204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.746480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.747212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] RBAC: Adding query filter params (role.id=a6d2f4a8b54a44b29b129335ac106b7b, user.id=1d8ce75cc349406fa01e3dd8f7ab2603, scope.system=all) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:47:59.747527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:47:59.749445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:47:59.764401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 193/375] 10.222.0.46 () {56 vars in 1308 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/role_assignments?user.id=1d8ce75cc349406fa01e3dd8f7ab2603&scope.system=all&role.id=a6d2f4a8b54a44b29b129335ac106b7b => generated 510 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:47:59.765347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:47:59.874875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458851c5-07a2-4c17-a7e6-ce811e41149d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:47:59.967162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:47:59.983696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:47:59.984335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:47:59.984619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:47:59.984896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:47:59.985793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d733f-b707-49f7-be14-3410fecf3b78 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:00.047421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:00.056671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:00.057626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:00.058183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:00.058663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:00.059613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=519f588961584ff99d94b1382275b6e9) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:00.060132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:00.062052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:00.074572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 194/376] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=1126c16f80da4dde8de7192a7858171e => generated 260 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:00.094043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:00.095549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:00.208909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 183/377] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:47:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 601 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:00.214567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:00.369865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1b3a3-2af9-4df3-a1ac-179de8cc0a35 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:00.408374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:00.409371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:00.409853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:00.410347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] PATH_INFO: `/v3/projects/2b351b41bbb64be2822e6354d987b945` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:00.411681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] RBAC: Authorizing `identity:get_project(project_id=2b351b41bbb64be2822e6354d987b945)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:00.413375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:00.415806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 195/378] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:00 2022] GET /identity/v3/projects/2b351b41bbb64be2822e6354d987b945 => generated 344 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:00.421643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:00.502986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:00.553596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8e1a0-42da-463e-911e-91517f532b9b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:00.597471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:00.598126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:00.598406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:00.598679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] PATH_INFO: `/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:00.606176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] RBAC: Authorizing `identity:get_group(group_id=92e5cdfe8de14723bbca8a9e7eb8b2ec)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:00.607686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f2d02-2a7f-48f4-9a9e-f4833243d738 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:00.614310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 184/379] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:00 2022] GET /identity/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec => generated 229 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:00.615623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:00.661455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:00.751391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af1ace-1091-448b-b13f-4f9d775d2c82 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:00.792688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:00.793228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:00.793506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:00.793778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:00.794537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] RBAC: Adding query filter params (group.id=92e5cdfe8de14723bbca8a9e7eb8b2ec, role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=2b351b41bbb64be2822e6354d987b945) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:00.794843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:00.796274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fee96d0-ad20-48c2-9951-c67a26d91cfc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:00.803138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 196/380] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:00 2022] GET /identity/v3/role_assignments?group.id=92e5cdfe8de14723bbca8a9e7eb8b2ec&scope.project.id=2b351b41bbb64be2822e6354d987b945&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:01.324146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8ab39-c9ec-44e7-a111-5d40eabbc552 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:01.324479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8ab39-c9ec-44e7-a111-5d40eabbc552 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:01.324715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8ab39-c9ec-44e7-a111-5d40eabbc552 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:01.326379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 185/381] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:01.331608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28e21-a1f8-43c6-b41a-e90de0cf1c1f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:01.331889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28e21-a1f8-43c6-b41a-e90de0cf1c1f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:01.332156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d28e21-a1f8-43c6-b41a-e90de0cf1c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:01.356272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-01d28e21-a1f8-43c6-b41a-e90de0cf1c1f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:01.357469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-01d28e21-a1f8-43c6-b41a-e90de0cf1c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:01.434062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 197/382] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:01.793251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7dd8a-774b-44af-930c-35e8464b73ab None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:01.793794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7dd8a-774b-44af-930c-35e8464b73ab None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:01.793794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7dd8a-774b-44af-930c-35e8464b73ab None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:01.795132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 186/383] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:01.800468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:01.801210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:01.801632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:01.824390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:01.825784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:01.846542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:01.848184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7dd8a-774b-44af-930c-35e8464b73ab None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:01.934032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 198/384] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:02.009804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7dd8a-774b-44af-930c-35e8464b73ab None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.042496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.059513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.060469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.060994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.061463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.081829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:02.087633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:02.088551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:02.089380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:02.094093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 187/385] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:02.097168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.236713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.238503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca778c-5c21-452e-b1a7-3759b4def09a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.295061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.295683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.295964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.296239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.296917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:02.297225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:02.298643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acfe52f-f7b6-4f80-8a76-98642614811b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:02.309382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 199/386] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:02.314777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26af2a7e-68b3-4970-a045-a69a1ac935f0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.315009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26af2a7e-68b3-4970-a045-a69a1ac935f0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.315292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26af2a7e-68b3-4970-a045-a69a1ac935f0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.316657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 200/387] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:02 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:02.322613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26af2a7e-68b3-4970-a045-a69a1ac935f0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.340745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.371365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c606182-0d89-4726-963f-a96d28b55a01 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.408384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.409233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.409517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.409795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.424817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fa50b3d5-79f3-49fe-9f91-1c001cfb94d0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:02.433690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:02.434618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:02.435412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 188/388] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:02.436069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.436896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.436896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.463071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:02.464292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:02.466423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26af2a7e-68b3-4970-a045-a69a1ac935f0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.509798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.511587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.513371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.513901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.517613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.517613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:02.518205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:02.520878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 201/389] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:02.522499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.555542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 189/390] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:02.557138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.645313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.675459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399d6cf-658d-4ffc-92e1-72fcd76b1988 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.710399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.711004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.711285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.711551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.712262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:02.712572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:02.713914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ffcb45-0822-41e1-9bd8-b180f2201c87 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:02.719848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca2ab8e-8447-40e1-9848-755b19adc2fd None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.722330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 202/391] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:02.723572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b6a1-f117-4a23-a855-de69c3a39577 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.723916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b6a1-f117-4a23-a855-de69c3a39577 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.724254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b6a1-f117-4a23-a855-de69c3a39577 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.725577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 203/392] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:02.730016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12b6a1-f117-4a23-a855-de69c3a39577 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.756504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.757135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.757995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.758253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.759365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:02.760788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117022d1-74f5-4abf-8c8c-0bf9f43a115e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:02.762687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 190/393] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:02.763783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.764065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.764338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.790683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:02.792535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:02.839105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.847095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12b6a1-f117-4a23-a855-de69c3a39577 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:02.875943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 191/394] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:02.882173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.894068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:02.894869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:02.895159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:02.895790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:02.896798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:02.898728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:02.900571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 204/395] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:02.901324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:02.914570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:02.988053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4ca86-1256-4404-8e7a-cca65f53cc19 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.019420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4544665-f8f4-462a-a6ec-0a1de81f3cb8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.035621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.036561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.036848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.037511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/groups/602ef0530de2472b8baf0e871244f76d/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.040019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, group_id=602ef0530de2472b8baf0e871244f76d, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.042175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.056846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.057526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.057801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.058140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.059728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 192/396] 10.222.0.46 () {58 vars in 1442 bytes} [Fri Oct 28 10:48:02 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/groups/602ef0530de2472b8baf0e871244f76d/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:03.062931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.087361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:03.099115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:03.100108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:03.101879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 205/397] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:03.108674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.135624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.220663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.253372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433356b-d1de-4483-b96c-a802c4ccaf49 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.259674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50b65b7-c1d7-44e7-b95f-99fb73fadfd1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.299812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.300694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.301128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.301546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.302831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.304834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.307718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 193/398] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:03.307718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.311786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.313323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.313614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.313897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.314647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:03.314957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.316342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.328192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 206/399] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:03.329086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.429773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a155493-127e-47fa-adf4-fcd6b8a165b1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.463132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368565d-125e-4993-bc8c-f514e8e2e91c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.466409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.475794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.477306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.477660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.478239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.491495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:03.497449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:03.497945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:03.498873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:03.499659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 194/400] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:03.500695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.508140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.509367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.509367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.509367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/groups/602ef0530de2472b8baf0e871244f76d/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.511529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, group_id=602ef0530de2472b8baf0e871244f76d, role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.512970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c7d3e0-3c59-4f46-88b7-fa7f89690fb1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.526642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 207/401] 10.222.0.46 () {58 vars in 1442 bytes} [Fri Oct 28 10:48:03 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/groups/602ef0530de2472b8baf0e871244f76d/roles/1126c16f80da4dde8de7192a7858171e => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:03.536497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b0f97-558a-4b38-a042-afcc2c4fbc2f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.537389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b0f97-558a-4b38-a042-afcc2c4fbc2f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.537897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172b0f97-558a-4b38-a042-afcc2c4fbc2f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.541844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 208/402] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:03 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:03.543063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172b0f97-558a-4b38-a042-afcc2c4fbc2f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.622621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc07e98-6e57-4834-aa7c-1bde100ae60f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.668437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.669576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.670163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.670638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.672106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.673848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34d5d22-3fb5-4589-bed9-66c2d024c9a9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.676299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 195/403] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:03.677902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.678436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.679014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.688551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172b0f97-558a-4b38-a042-afcc2c4fbc2f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.699914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:03.701549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:03.725914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.734476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.735784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.736639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.737372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.738359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:03.738856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:03.740600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:03.753155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 209/404] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:03.754006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.795085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 196/405] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:03.797012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:03.835450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:03.923364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8442d-0869-40a4-ad0a-ec158cc0891d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.927331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e3fe67-80c6-4978-92c4-95bcec04757f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:03.994069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.995472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:03.996395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:03.997180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:03.998982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:03.999771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.000549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.002176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.002621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.003043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.004501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.008078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 210/406] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:04.008403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.010216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 197/407] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:03 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:04.013167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.016773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.062402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:04.130899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1697d9-0fee-431c-93bb-113ae186518a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.140510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2198447-702d-40c0-ad23-56b91b13b056 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.163075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:04.181121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.181792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.182105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.182388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb/groups/602ef0530de2472b8baf0e871244f76d/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.184493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] RBAC: Authorizing `identity:create_grant(project_id=027d2665b26c4993a084d20e698ac8eb, group_id=602ef0530de2472b8baf0e871244f76d, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.186657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.194481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.195408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.195912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.196406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.197862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.199567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.201412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 211/408] 10.222.0.46 () {58 vars in 1442 bytes} [Fri Oct 28 10:48:04 2022] PUT /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb/groups/602ef0530de2472b8baf0e871244f76d/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 10:48:04.202162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 198/409] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:04 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:04.202414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.215083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.342027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffeab17-2961-4a2f-9784-4083a03a4697 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.377277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.377937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.378244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.378520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.390348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:04.396728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:04.397591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:04.398957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 212/410] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:04.405599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.406939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:04.426169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b747040-1674-4add-a61b-e4fc89be91c7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.459297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:04.471557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.472565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.473492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.474331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9/groups/602ef0530de2472b8baf0e871244f76d/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.477511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] RBAC: Authorizing `identity:create_grant(project_id=519f588961584ff99d94b1382275b6e9, group_id=602ef0530de2472b8baf0e871244f76d, role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.479630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.499062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 199/411] 10.222.0.46 () {58 vars in 1442 bytes} [Fri Oct 28 10:48:04 2022] PUT /identity/v3/projects/519f588961584ff99d94b1382275b6e9/groups/602ef0530de2472b8baf0e871244f76d/roles/1126c16f80da4dde8de7192a7858171e => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:04.539884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf9374-b6d8-4bb0-a32a-41689f72fbcd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.581100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.581739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.582042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.582320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.583181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:04.583554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.585153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.596051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 213/412] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:04.601563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.742150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2339b759-e419-45e9-8b0b-aaa9be883991 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.782477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.783459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.783933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.784500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] PATH_INFO: `/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.785899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] RBAC: Authorizing `identity:get_group(group_id=92e5cdfe8de14723bbca8a9e7eb8b2ec)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.787579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.789662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 200/413] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:04 2022] GET /identity/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec => generated 229 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:04.794929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:04.940687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc2b845-ddcc-4793-a33d-71c3d4fc8ee5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:04.943100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:04.993569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:04.994292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:04.994573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:04.994844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] PATH_INFO: `/v3/projects/2b351b41bbb64be2822e6354d987b945` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:04.995980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] RBAC: Authorizing `identity:get_project(project_id=2b351b41bbb64be2822e6354d987b945)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:04.997416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c6e651-78c8-4aa4-b08d-f65c3c621ff5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:04.999356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 214/414] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:04 2022] GET /identity/v3/projects/2b351b41bbb64be2822e6354d987b945 => generated 344 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:05.005025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:05.031234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:05.118787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966c1951-d989-469d-96a9-de979d68ba59 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:05.167652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:05.168339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:05.168624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:05.168901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] PATH_INFO: `/v3/projects/2b351b41bbb64be2822e6354d987b945/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:05.170871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] RBAC: Authorizing `identity:create_grant(project_id=2b351b41bbb64be2822e6354d987b945, group_id=92e5cdfe8de14723bbca8a9e7eb8b2ec, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:05.172339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b142c5-90fe-4464-bff6-efbdc03f1dd6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:05.202464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 201/415] 10.222.0.46 () {58 vars in 1442 bytes} [Fri Oct 28 10:48:05 2022] PUT /identity/v3/projects/2b351b41bbb64be2822e6354d987b945/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:05.650755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c29886-14b4-41c6-b7ed-76b378817d36 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:05.651440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c29886-14b4-41c6-b7ed-76b378817d36 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:05.651918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c29886-14b4-41c6-b7ed-76b378817d36 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:05.653825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 215/416] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:05.658801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf083c-93ab-44e8-bac2-160674ef4d24 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:05.659307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf083c-93ab-44e8-bac2-160674ef4d24 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:05.659799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf083c-93ab-44e8-bac2-160674ef4d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:05.680945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-01cf083c-93ab-44e8-bac2-160674ef4d24 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:05.682575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-01cf083c-93ab-44e8-bac2-160674ef4d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:05.771073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 202/417] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:06.200621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c29886-14b4-41c6-b7ed-76b378817d36 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:06.242352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60872508-b13a-4c66-9043-9ea39c83af0e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.242851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60872508-b13a-4c66-9043-9ea39c83af0e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.242949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60872508-b13a-4c66-9043-9ea39c83af0e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.244598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 203/418] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:06.265562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.265876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.266165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.295888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:06.297087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:06.307712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:06.347122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c29886-14b4-41c6-b7ed-76b378817d36 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:06.377857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 204/419] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:06.412553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:06.413207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.414067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.414350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.431076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:06.436028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:06.436890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:06.437671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 216/420] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:06.444658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:06.586592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2009c149-3fa2-4bdf-b517-d2e33a86fc26 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:06.643150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:06.643772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.644049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.644317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.644984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:06.645284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:06.646683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:06.658093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 205/421] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:48:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:06.663800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:06.669508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:06.672328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:06.701055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:06.840566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff142b-be67-47fc-a00c-bc39585d101d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:06.867962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26603c19-9859-4b9f-b48e-a24c979f66c0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:06.891282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:06.892500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.893376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.894227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.904885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:06.905532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.906159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.906159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.917547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-40456e1b-1050-46b6-bab6-b60087d53f0f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:06.923652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:06.924543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:06.925355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 206/422] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:06.926565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9aba4-bc5f-44ca-95df-888c9e990996 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:06.926650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9aba4-bc5f-44ca-95df-888c9e990996 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:06.926895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c9aba4-bc5f-44ca-95df-888c9e990996 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:06.932311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 207/423] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:06.932311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c9aba4-bc5f-44ca-95df-888c9e990996 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:06.991785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:06.994376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:06.995199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.028769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.054053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c9aba4-bc5f-44ca-95df-888c9e990996 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.091588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.092410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.092539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.092718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.093434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:07.093743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.095186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b708154-7eae-4327-ac9c-11208a34599a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.103845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 208/424] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:07.105447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.105739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.106032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.126886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:07.126886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:07.127500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 217/425] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:07.129528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.202145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 209/426] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:07.203051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.271967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ac07-5cb0-4cf2-a900-97061029acf2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.305935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.318619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.319555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.320610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.321103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.342631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.391323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00696f9-c84c-4867-9dad-12ac4ca155b0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.422703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.424400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04309bca-60a1-4286-87fb-fad78ee762bb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.426683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.428019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.428019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.428019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.429900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.431751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740b85f8-4369-4d98-b287-6f5fa99f62f8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.433830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 210/427] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:07.438278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979c9cf-ec74-4e94-859a-fb3616ab894f None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.438278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979c9cf-ec74-4e94-859a-fb3616ab894f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.438278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979c9cf-ec74-4e94-859a-fb3616ab894f None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.441095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 211/428] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:07.442027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b979c9cf-ec74-4e94-859a-fb3616ab894f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.547512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 218/429] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:07.549168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.549436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.549807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.563670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b979c9cf-ec74-4e94-859a-fb3616ab894f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.569158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:07.570502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:07.589300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.594968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.603337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.603913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.604450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.604727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.605849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.607345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.608908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 212/430] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:07.613759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.645039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 219/431] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:07.645753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.755522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9aed23-7777-429a-b2f2-eb842e4e0188 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.762643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec1e2d2-4b07-46f5-bef8-2ea024d557a7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.790676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.791312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.791588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.791859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.799035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.799671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.799953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.800227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.801341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.802525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:07.802821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.809544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 220/432] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:07.809544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7dc44b74-addd-4266-935a-c5991b38da60 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:07.809742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:07.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 10:48:07.811321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:07.814119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 213/433] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:07.815749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a06f4c-ba6e-4a0e-bd2f-0ca8b4006729 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.816037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a06f4c-ba6e-4a0e-bd2f-0ca8b4006729 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.816314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a06f4c-ba6e-4a0e-bd2f-0ca8b4006729 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.818013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 214/434] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:07.818325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a06f4c-ba6e-4a0e-bd2f-0ca8b4006729 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.842254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.877634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:07.919298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c63c79-d840-46f2-8899-dbe7c1fb50fa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.931440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a06f4c-ba6e-4a0e-bd2f-0ca8b4006729 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:07.952069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.952683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.952959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.953228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.953944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=770c965de4374724a8a6de8042997c97, scope.project.id=519f588961584ff99d94b1382275b6e9) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:07.954274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.955694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.961894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 221/435] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=770c965de4374724a8a6de8042997c97 => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:07.962686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:07.965602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:07.966292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.966588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.966849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.968076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:07.969579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e5ac6-6aa0-4432-a7c4-495081369b7a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:07.971281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 215/436] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:07.972606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:07.972891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:07.973165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:07.993127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:07.994534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:08.067828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 216/437] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:08.069153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.078892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.086087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5260b40-2e32-41b6-8434-918225e0363c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.125845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.126500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.126775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.127048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.127802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] RBAC: Adding query filter params (name=member) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:08.128012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.129384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.139364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 222/438] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:48:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:08.140191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.141498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.206859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2a23a2-3e59-4131-9472-ec3ca2785c16 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.252474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.253124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.253402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.253726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.264934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a87628-76bc-48b6-a8b2-233787333898 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.265919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:08.272621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:08.274122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:08.275102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:08.277831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 217/439] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:08.279558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.306808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.307474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.307762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.308085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.308852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=027d2665b26c4993a084d20e698ac8eb) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:08.309218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.310750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.322392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 223/440] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=1126c16f80da4dde8de7192a7858171e => generated 611 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:08.323692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.352103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.402313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb54699-dceb-496d-ac06-83dd877bd28d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.413275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.442899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.443855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.444268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.444809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.446491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b31c50-676e-4ae9-ab87-c73c60cb22e5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.485037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.485655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.485932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.486343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.486892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:08.487190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.488548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.498179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 224/441] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:08.499091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.536527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.540573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.588639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.608818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cbb550-3758-46be-ab7c-5463a4ef29e3 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.638332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 218/442] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:08.643230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.645907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.646567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.646849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.647120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.651614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.658042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:08.663319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:08.664735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 225/443] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:08.666357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.744064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b15821e-0824-4307-96a4-ac7fda243404 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.763424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5790c28d-08dc-43c7-a6da-a54d86175ac0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.776661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.777261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.777534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.777802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.796592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.797206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.797483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.797752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] PATH_INFO: `/v3/projects/027d2665b26c4993a084d20e698ac8eb` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.798897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] RBAC: Authorizing `identity:get_project(project_id=027d2665b26c4993a084d20e698ac8eb)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.800321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.801914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 226/444] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/projects/027d2665b26c4993a084d20e698ac8eb => generated 298 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:08.802695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.812857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.847040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.848530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.858863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:08.901789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc654a89-2e15-46dd-95d5-4b8c0d29fff7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:08.934355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:08.934888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:08.935166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:08.935444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:08.936093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:08.936392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:08.937737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:08.949909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 219/445] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:08.950102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 227/446] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:08.950182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:08.951039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.018443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.043147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2b5094-1418-4e6e-bbd7-34a5d859ee6d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.051503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd44e0-bc88-4c0e-9fc8-e37ba77fe7ca None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.052289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.077539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.078187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.078469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.078744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.079853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.081289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.083057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 220/447] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.083724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.084649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.085859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.086173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.086449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.153095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.154608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.180149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f98d4e-a839-4b69-b7c8-cd37be1f09ca None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.214548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.215094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.215369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.215636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] PATH_INFO: `/v3/projects/519f588961584ff99d94b1382275b6e9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.216735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] RBAC: Authorizing `identity:get_project(project_id=519f588961584ff99d94b1382275b6e9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.218192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.219772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 221/448] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/projects/519f588961584ff99d94b1382275b6e9 => generated 302 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.220515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.220979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.256751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 228/449] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:09.258478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.266373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.321035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0437c-ca08-472c-85d8-49fc911e1747 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.353814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.354694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.354975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.355245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.355982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=770c965de4374724a8a6de8042997c97, scope.project.id=027d2665b26c4993a084d20e698ac8eb) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:09.356274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.357713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.359258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e16ad-1fc5-45ed-a21c-33cc538ffe55 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.363732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 222/450] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=027d2665b26c4993a084d20e698ac8eb&role.id=770c965de4374724a8a6de8042997c97 => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.364505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.400315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.400978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.401261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.401535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] PATH_INFO: `/v3/groups/602ef0530de2472b8baf0e871244f76d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.402756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] RBAC: Authorizing `identity:get_group(group_id=602ef0530de2472b8baf0e871244f76d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.404246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.406129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 229/451] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/groups/602ef0530de2472b8baf0e871244f76d => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.411901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.455665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.484431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7575db-914e-4205-bb1c-8c9c85c0494b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.496311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.526423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.526686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618883fb-2c22-4dab-8a94-48e06a302b8b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.527081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.527363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.527639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] PATH_INFO: `/v3/projects/2b351b41bbb64be2822e6354d987b945` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.528876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] RBAC: Authorizing `identity:get_project(project_id=2b351b41bbb64be2822e6354d987b945)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.530379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.532035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 223/452] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/projects/2b351b41bbb64be2822e6354d987b945 => generated 344 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.537485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.572285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.572927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.573171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.573448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.574214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] RBAC: Adding query filter params (group.id=602ef0530de2472b8baf0e871244f76d, role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=519f588961584ff99d94b1382275b6e9) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:09.574525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.575991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.582650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 230/453] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/role_assignments?group.id=602ef0530de2472b8baf0e871244f76d&scope.project.id=519f588961584ff99d94b1382275b6e9&role.id=1126c16f80da4dde8de7192a7858171e => generated 611 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:09.655844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23914f81-4d5c-468d-b848-712419e45bf6 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.693443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.694127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.694410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.695235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] PATH_INFO: `/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.696535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] RBAC: Authorizing `identity:get_group(group_id=92e5cdfe8de14723bbca8a9e7eb8b2ec)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.698073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ceaaee-1164-45e5-bb5b-68119d8d3089 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.700015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 224/454] 10.222.0.46 () {56 vars in 1180 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/groups/92e5cdfe8de14723bbca8a9e7eb8b2ec => generated 229 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:09.706635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:09.819357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab701e42-f294-428b-a28e-c930e9ed0993 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:09.820871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:09.854519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:09.855144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:09.855424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:09.855698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:09.856423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] RBAC: Adding query filter params (group.id=92e5cdfe8de14723bbca8a9e7eb8b2ec, role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=2b351b41bbb64be2822e6354d987b945) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:09.856735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:09.858221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2b96-fcbc-49aa-9832-960f81ee45ac None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:09.864158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 231/455] 10.222.0.46 () {56 vars in 1376 bytes} [Fri Oct 28 10:48:09 2022] GET /identity/v3/role_assignments?group.id=92e5cdfe8de14723bbca8a9e7eb8b2ec&scope.project.id=2b351b41bbb64be2822e6354d987b945&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 611 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:12.525363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d0947-b68f-4c44-ad9a-757ef061bced None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.526351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d0947-b68f-4c44-ad9a-757ef061bced None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.526827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9d0947-b68f-4c44-ad9a-757ef061bced None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:12.528372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 225/456] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:12.533832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025e0517-a175-451b-a1fb-3e3c4b7e17c3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.534644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025e0517-a175-451b-a1fb-3e3c4b7e17c3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.535108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025e0517-a175-451b-a1fb-3e3c4b7e17c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:12.558717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-025e0517-a175-451b-a1fb-3e3c4b7e17c3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:12.560232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-025e0517-a175-451b-a1fb-3e3c4b7e17c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:12.650528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 232/457] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:12.891205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cc94d-36ba-442a-913d-5464eed76eb8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.891880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cc94d-36ba-442a-913d-5464eed76eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.892351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cc94d-36ba-442a-913d-5464eed76eb8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:12.894167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 226/458] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:12.899686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.900188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.900655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:12.921001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:12.922515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:12.973598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a86cc-2697-42e5-a610-e6a594c3ca37 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.974247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a86cc-2697-42e5-a610-e6a594c3ca37 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.974714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8a86cc-2697-42e5-a610-e6a594c3ca37 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:12.976265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 227/459] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:12.981614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:12.981896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:12.982196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.014853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:13.016058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:13.028803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 233/460] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:13.062417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.098582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.110316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.155343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 228/461] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:48:13.206606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d50868-a8c4-4934-b1e0-2e6cf3a729df None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.245396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.251560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.252052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.252519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.276124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.277875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:13.300069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:13.300979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:13.301822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:13.301822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.301822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:13.301822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 234/462] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:13.311048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.357090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.492648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a5c54f-0af2-40d6-83f4-5440a50d45b5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.493700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.520713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795d77f-5989-4387-88a7-175815c09e1c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.536818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.537425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.537707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.538002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.540117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:13.540423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.541799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b880e3e-4beb-49c0-b9c2-0c96c91556a0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:13.553375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 229/463] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:13.555149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21590899-f463-4c23-9565-3e6d90ac8119 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.555435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21590899-f463-4c23-9565-3e6d90ac8119 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.555708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21590899-f463-4c23-9565-3e6d90ac8119 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.556897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 230/464] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:13.557582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21590899-f463-4c23-9565-3e6d90ac8119 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.562940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.563701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.564108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.564383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.576311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.577762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:13.592452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:13.593424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:13.594301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:13.594301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.594301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:13.594301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 235/465] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:13.594510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.603215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.674482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21590899-f463-4c23-9565-3e6d90ac8119 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.706576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb8cce5-3e6e-4136-8d67-09077bc7f1d9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.709351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.710030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.710873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.711152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.722619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.724069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:13.737248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-41a3e7b7-a1ce-402c-b704-d3db34bc44af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:13.739333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:13.740846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:13.741688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:13.741688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:13.741688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:13.741688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 231/466] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:13.742469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.742804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.743027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.749555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.750208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.750690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.750966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.752105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.753041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:13.762380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:13.763568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:13.785998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 236/467] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:13 2022] POST /identity/v3/users => generated 246 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:13.785998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.848053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 232/468] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:13.849022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:13.884834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:13.954438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179f0e8a-5bde-4107-a682-eeee86e53731 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:13.991200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:13.992930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:13.993448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:13.993920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:13.994860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:13.995495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:13.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.012367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 237/469] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:14.018267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.033789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d4850b-ed38-4c8a-9c1e-af5366f791b7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.071489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:14.074298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.075245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.075703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.076179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.077002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:14.077522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.079261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.096238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 233/470] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:14.101268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.137632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cd16d-eb46-4fde-b648-ee1b6b6178a5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.183743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.184681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.185141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.185589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.186445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.188061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b226690b-4221-40c4-ad44-92e097c99f89 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.211585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 238/471] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:14 2022] POST /identity/v3/users => generated 248 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:48:14.212970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be987c23-071d-4e60-9755-39bf499f76c4 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.213264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be987c23-071d-4e60-9755-39bf499f76c4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.213545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be987c23-071d-4e60-9755-39bf499f76c4 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.218686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 239/472] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:14.218686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be987c23-071d-4e60-9755-39bf499f76c4 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.223043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:14.252885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe432c3-2545-4696-9855-e67cda112386 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.301040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.302541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.303065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.303554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.305452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.308062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8e0f1c-053e-46e6-bf91-db5392e45506 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.337883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 234/473] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:14 2022] POST /identity/v3/users => generated 248 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:14.339870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.340650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.341101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.371193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be987c23-071d-4e60-9755-39bf499f76c4 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.372119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:14.373602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:14.376376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:14.426479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.427424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.427710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.427984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.442069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.443286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:14.462993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:14.464151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:14.465182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:14.465182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:14.465182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.465457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 240/474] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:14.471892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.497903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 235/475] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:14.596359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b28a9-b5f0-4f01-adaa-ca207d61f94d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.633671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.634550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.634978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.635420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.636221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:14.636682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.638305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.652551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 241/476] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:14.657678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.747218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:14.784830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:14.796656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faa8e24-6146-415d-b020-13f122e4a09a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.806844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:14.845693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.846343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.846673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.846981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.847641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.849080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.894329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 236/477] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:14 2022] POST /identity/v3/users => generated 249 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:48:14.907979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3457d6-8e4b-4629-a7c4-9850de4c1599 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:14.945829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:14.946503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:14.946783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:14.947053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:14.947760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:14.949171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:14.959072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.sql.core [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 10:48:14.959072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 10:48:14.959072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [parameters: {'id': 'c8747663063f4f4b8341801997a6f21f', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 10:48:14.959072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81305) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result.read() Oct 28 10:48:14.977165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return ref.to_dict() Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:48:14.978128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application yield resource Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise self.value Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application session.commit() Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 10:48:14.979724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.session.flush() Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application compat.raise_( Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 10:48:14.981583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:48:14.986155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result.read() Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:48:14.988283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [parameters: {'id': 'c8747663063f4f4b8341801997a6f21f', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:14.990498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 10:48:14.992646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:14.994692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 242/478] 10.222.0.46 () {60 vars in 1131 bytes} [Fri Oct 28 10:48:14 2022] POST /identity/v3/roles => generated 144 bytes in 194 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:14.994828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:15.142451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f172bd-416b-4316-9f48-e43e39098d51 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:15.161805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:15.181572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:15.182255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:15.182563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:15.182811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:15.193599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-a30ad8a0-d991-4596-bea0-b79b56c9d03f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:15.199521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:15.200367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:15.201178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 237/479] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:15.206710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:15.295353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:15.319205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b17df1-11d4-4718-8ba3-0a60075640f0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:15.358000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:15.358673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:15.358958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:15.359237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:15.359982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:15.360287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:15.361664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3aa85c-f70b-4ba8-9c28-b6a1d1f0ba8c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:15.374481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 243/480] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:16.691004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81bcfc-72e0-4a24-a808-9e45552496e4 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:16.691514 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81bcfc-72e0-4a24-a808-9e45552496e4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:16.691514 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81bcfc-72e0-4a24-a808-9e45552496e4 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:16.693106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 238/481] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:16.698853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329eda7-05f4-47e6-9ee7-3a21b279c5d2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:16.699278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329eda7-05f4-47e6-9ee7-3a21b279c5d2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:16.699445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6329eda7-05f4-47e6-9ee7-3a21b279c5d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:16.718308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6329eda7-05f4-47e6-9ee7-3a21b279c5d2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:16.719783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6329eda7-05f4-47e6-9ee7-3a21b279c5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:16.789757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 244/482] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:17.108309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81bcfc-72e0-4a24-a808-9e45552496e4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.126266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ddc4f-8edd-4bab-a88e-7e40c1ad5375 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.126913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ddc4f-8edd-4bab-a88e-7e40c1ad5375 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.127385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62ddc4f-8edd-4bab-a88e-7e40c1ad5375 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.128925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 245/483] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:17.143583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65019600-9c28-40c7-bd71-b5138a32e252 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.144178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65019600-9c28-40c7-bd71-b5138a32e252 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.144652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65019600-9c28-40c7-bd71-b5138a32e252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.166325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-65019600-9c28-40c7-bd71-b5138a32e252 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:17.168453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-65019600-9c28-40c7-bd71-b5138a32e252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:17.244955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81bcfc-72e0-4a24-a808-9e45552496e4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.245841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:17.255537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 246/484] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:48:17.255537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de44d4d-2503-4c46-b94f-8a3e4bb1182c None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.255537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de44d4d-2503-4c46-b94f-8a3e4bb1182c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.255537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de44d4d-2503-4c46-b94f-8a3e4bb1182c None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.255537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 247/485] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:17.276057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.276057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.276057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.303786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:17.304978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:17.325016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.325648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.325923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.326212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.345873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:17.352912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:17.353769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:17.354857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 239/486] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:17.362378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.435298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 248/487] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:17.435661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:17.509766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b98d414-8e2f-4a87-af6e-3eb720e12d80 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.543693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.544345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.544626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.544893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.545554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:17.545849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:17.547261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:17.556433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 240/488] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:17.562348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.611073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.726303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f85925-4583-4f69-8d64-4a7609dbee1e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.730363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafb2850-ba07-459f-8fe5-e9103c22a0a3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.746416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:17.762602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:17.769310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.770178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.770459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.770796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.771089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.771356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.771461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.771626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.783231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:17.784680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:17.785079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-49d34d58-8c7b-4bf1-8452-43364d4c97f7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:17.791071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:17.791952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.792755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:17.794134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 241/489] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:17.794134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71675959-f789-4d0c-a0e4-7bdbdd5a82f8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.794430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71675959-f789-4d0c-a0e4-7bdbdd5a82f8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.795452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71675959-f789-4d0c-a0e4-7bdbdd5a82f8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.796639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 242/490] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:17.797341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71675959-f789-4d0c-a0e4-7bdbdd5a82f8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:17.803430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:17.804854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:17.805682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:17.805682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:17.805682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:17.805682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 249/491] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:17.806626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.904508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71675959-f789-4d0c-a0e4-7bdbdd5a82f8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.916202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fdebe2-7ce2-4720-a0e8-bc466ccb6fe0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:17.940612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.941498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.941998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.942499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.950962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:17.951571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.951853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.952127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.952796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:17.953103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:17.954539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ff92d-5aad-4af5-aecc-d946d96bf7ba None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:17.955163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:17.960464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:17.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:17.962136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:17.962726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 250/492] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:17.963834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:17.964148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:17.964425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:17.964570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 243/493] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:17.965434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:17.984437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:17.985536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:17.994387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.030647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.066115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 251/494] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:18.067478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.073498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8d72dc-842d-40a8-9f5d-e775c434b917 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.108088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.108759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.109117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.109294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.109934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:18.110269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.111714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.129576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 244/495] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:18.129576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.207482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598748ba-6354-445d-b7d9-1ee0cd158a21 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.234397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.246229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.246930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.247311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.247909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.249540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad49d02-f2ad-4662-9160-dbc67a9fd4b8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.259523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.261175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:18.276983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:18.277858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:18.279008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:18.279008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:18.279008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:18.282282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 252/496] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:18.283320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.292232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.293270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.293542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.293814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.294868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:18.295179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.296853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080facc4-3827-4f8b-b06e-9243a98b3a77 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.308206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 245/497] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:18.309629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029caeb-7096-4696-b23f-99ccf5362a64 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.309995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029caeb-7096-4696-b23f-99ccf5362a64 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.310350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029caeb-7096-4696-b23f-99ccf5362a64 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.314026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 246/498] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:18.314026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e029caeb-7096-4696-b23f-99ccf5362a64 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.317994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.394915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c4c8a4-e305-4351-b841-a992798b593c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.412923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e029caeb-7096-4696-b23f-99ccf5362a64 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.428107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.428807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.429093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.429366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.436625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.438089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.445627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.446247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.446526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.446793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.447419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:18.447717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.449054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3003a56-0543-4609-b4be-a1aeaec9188d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:18.450513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:18.451544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:18.452624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 253/499] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:18.453291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.463475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 247/500] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:18.464755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.465099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.465448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.471741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.484950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:18.486257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:18.543356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.550860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 248/501] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:18.551583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.573003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec7781c-9be1-40c6-bbc6-1f965fa03360 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.607449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.608256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.608632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.609009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.619934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.621382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:18.637028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:18.638106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:18.638944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:18.638944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:18.638944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:18.638944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 254/502] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:18.638944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.659911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40a2624-d47c-4b34-b108-70500b485637 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.692291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.699186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.699930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.700355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.700629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.711302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:18.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:18.717025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:18.717818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 249/503] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:18.719010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.748748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465882f-ed37-4a23-adea-2276c8eb1fc5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.782174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.782780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.783059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.783338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.784059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:18.784363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.785797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.803613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.823149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 255/504] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:18.823981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.830379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23763a85-9791-4a87-b824-531c56215e12 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.865674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.867204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.867485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.867759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.874571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.876058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:18.888608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:18.889504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 250/505] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/users/cinder => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:18.890961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.926254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208dd3-ce65-495b-aad7-809107f39d29 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:18.945081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:18.962026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:18.962636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:18.962916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:18.963185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:18.963826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:18.964125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:18.965499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:18.976613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 256/506] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:18.977401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:18.991633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ddbad8-3c8a-44bb-8c21-6297be89147a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.024489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.025085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.025354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.025626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.026335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:19.026637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.028017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.035141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.035554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 251/507] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:19.036303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.082398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115efd3-ab11-4888-9960-7a3f95a31d69 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.115608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.116323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.116605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.116876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.127169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.128610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.131349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65306b-0538-49c3-9ee8-2766f69bd741 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:19.142776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:19.143780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:19.144916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:19.144916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.144916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:19.144916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 257/508] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:19.144916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.150409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.170207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.170816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.171097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.171372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.236596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.238066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.242751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ad535d-0527-4832-9cb1-4f915af661fa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.251640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.277143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.277752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.278062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.278338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.279074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:19.279386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.280837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.297754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 258/509] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:19.298412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.339767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 252/510] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:19.340530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.354737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.401402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed88835-e477-4002-8edf-90b5c3dedba9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.436094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.436519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a56fce-61c1-47e5-8e13-62015fecb33c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.436725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.437006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.437288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.444340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.446334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:19.457631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:19.458750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 259/511] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/users/glance => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:19.459646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.474217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.474847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.475127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.475399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.486142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.487595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.487792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:19.502240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:19.503237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:19.504067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:19.504067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.504067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:19.504067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 253/512] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:19.504067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.563052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8592500-d57a-4358-beb0-9c9669e61829 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.567761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.596893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.597430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.597709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.598006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.598645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:19.598951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.600308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.605169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba8ba69-437e-4777-96fd-0f373746a196 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.611988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 260/513] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:19.612771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.638908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.639502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.639778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.640046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.699290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.705927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.707395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.714770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6121392f-786e-4197-a26c-6b8f1a153f0d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.748209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.748830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.749377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.749377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.759562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.760994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.768720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:19.776720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:19.777617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:19.778512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:19.778512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:19.778512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:19.778512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 261/514] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:19.778728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.804659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 254/515] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:19.805497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.879798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f33fa4f-c5ee-4dbc-ae3e-5ab032981a7b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.902411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b1dbdb-043b-4b7a-96b2-72b1c0d90612 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:19.914315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.914917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.915194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.915466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.916187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:19.916490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.917915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.926178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:19.936265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 262/516] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:19.937031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.937141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:19.937799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:19.938170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:19.938496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:19.939184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:19.939547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:19.940899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:19.952503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 255/517] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:19.953293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:19.982160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.038170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca41c2-5902-4886-84f1-102293b96d16 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.057467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974a20e0-ffcc-4de0-abdb-e734268ad1cc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.073398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.074034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.074320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.074593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.075238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.076712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.090033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.090627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.090897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.091167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.091797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:20.092100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.093447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.097200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 263/518] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:19 2022] POST /identity/v3/users => generated 247 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:20.098019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.105774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 256/519] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:20.106680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.168485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.209820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.226860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bced0-4661-424e-897f-3c726cbdf6b5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.228047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01d40bf-c410-4e41-aa9e-867acd4f3398 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.264974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.265642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.265928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.266233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.271898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.272001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.272203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.273003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.281212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.283900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:20.300547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:20.301456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:20.302359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:20.302359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:20.302359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:20.302359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 264/520] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:20.302568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.352630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.354141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.411990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663aa833-88e8-45be-aa0c-99d67b77bbc8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.415359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.438502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.452285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.452925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.453204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.453480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.455239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 257/521] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:20.456063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.466969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.468420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:20.480581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:20.481483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 265/522] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/users/neutron => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:20.483200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.563028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c081f34-3800-4bdb-8258-f9e7259d56c3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.597316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb4696f-864f-469c-a3fe-4c8b2301e7c8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.597668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.598347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.598624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.598903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.606567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.608017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:20.620721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:20.621611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:20.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 258/523] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:20.622926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.638902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.639519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.639801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.640075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.640716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:20.641130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.642422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.658895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 266/524] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:20.658895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.679277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.683836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.736370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33e83b9-452e-4243-a2b5-4e3901141ff2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.763965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5564c532-654f-4343-a77d-f50e490147bd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.773919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.774578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.774858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.775126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.786076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.787451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.798839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.799454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.799733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.800007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.800733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:20.801042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:20.801433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:20.802324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:20.803149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:20.803149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:20.803149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:20.803319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.807673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 259/525] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:20.808459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.822712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 267/526] 10.222.0.46 () {56 vars in 1139 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:20.823387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.908335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.910421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:20.915329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d65b9-ce76-47e3-897d-4fcce68247c3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.929551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045aa8c9-d24c-4c6c-9522-2d96804c0cd7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:20.950844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.951455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.951740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.952010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.952704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:20.953015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.954506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.964186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:20.964846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:20.965080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:20.965348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:20.977235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 260/527] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:20.977235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:20.978805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:20.980500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:20.992917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:20.993942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:20.994949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 268/528] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:20.995403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.092750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020cf75e-5eb6-404d-ae10-2350ae131072 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.105186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba79ad-b20c-42db-9004-0d397551ab9c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.128731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.129342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.129622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.129894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.130588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.130891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.132304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.141375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.142098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.142381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.142655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.143563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 261/529] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.148014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.158853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.166563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.217475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.219072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.260773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108be3d4-b4ae-4d44-9d08-67bdb16fba66 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.296998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.297601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.297877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.298178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.298899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.299208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.300639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.318201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 262/530] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.319180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.334991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 269/531] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:21.334991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.395394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.449185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.461892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89472785-cb69-47cc-adea-6b7e566a0661 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.481764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9bb1cf-8659-4dc9-b7c9-8c82b3072121 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.517832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.518620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.518902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.519179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.520265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.520878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.521154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.521639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.522427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.522741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.524180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.526332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.528140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.536670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 263/532] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.537677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:21.542158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:21.542981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:21.543789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 270/533] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:21.547747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.659212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d9eb74-a89c-4ff5-b065-6acbd2a6bcc3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.673247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104af6f-3b0a-44c5-aaad-dd96f46aef08 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.675996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.696434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.697193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.697543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.698086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.708418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.710241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.712227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:21.718197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.719416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.720037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.721472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.722915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=2297b30fe14d4de8a3f39e247ed59dfb) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.723399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.725005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:21.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:21.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:21.729638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:21.729638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:21.729638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:21.733905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 271/535] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/role_assignments?user.id=2297b30fe14d4de8a3f39e247ed59dfb&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.734043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 264/535] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:21.734982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.735212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.857896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63f3260-45c3-470c-a75e-79786691740e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.872558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606b345e-a501-4837-81c5-1203b2d45f42 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:21.900142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.901168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.901644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.902177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.903082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.903582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.905181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.920598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 272/536] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.927626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.946948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:21.947707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:21.947888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:21.948167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:21.948880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:21.949189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:21.950767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:21.967669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 265/537] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:21.974433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:21.980205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:22.081091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:22.092937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f805d9-cc16-43ee-bff3-be8c77fffcdb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:22.129776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:22.130372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.130651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.130920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.146117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:22.148141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:22.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:22.174796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:22.175615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:22.176952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32affc2f-c185-468a-97f9-d6f0734dd3e9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:22.179232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 273/538] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:22.185842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:22.220203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:22.220817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.221096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.221368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.222119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=589508bdaf3c4e159c01e3a43eb69392) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:22.222428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:22.223861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:22.230774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 266/539] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:21 2022] GET /identity/v3/role_assignments?user.id=589508bdaf3c4e159c01e3a43eb69392&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:22.342548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:22.362544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dde3ee-1efc-4203-a872-4f5f724883a8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:22.415237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:22.415783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.416063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.416330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.417042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:22.417334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:22.418819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fd2b7d-ab76-402d-bebc-9ccb7b9f6965 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:22.443500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 274/540] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:22.450370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:22.479580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3859630c-326e-449f-813e-09ba09b46fbb None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.480574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3859630c-326e-449f-813e-09ba09b46fbb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.481064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3859630c-326e-449f-813e-09ba09b46fbb None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.483512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 275/541] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:22.489286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.490301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.490935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.523520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:22.524705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:22.612996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31823f0-6eec-4efc-9c2b-744a4142b0a8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:22.641254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:22.654444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 276/542] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:22.671367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:22.671904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:22.672177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:22.672445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:22.673170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0287d4acc8f548c482ad40bd61b1081d) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:22.673477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:22.674931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:22.686414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 267/543] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:22 2022] GET /identity/v3/role_assignments?user.id=0287d4acc8f548c482ad40bd61b1081d&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:22.946141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.035983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:23.065783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a70ec1-2beb-4dc1-b3a3-6b05823594b1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:23.099741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:23.100369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.101258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.101531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.119584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:23.124338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:23.125176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:23.125940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 277/544] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:23.131460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.256235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd5b618-c19f-4bec-954c-db37414c9962 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:23.307592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:23.308250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.308532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.308796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.309461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:23.309761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:23.311152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:23.323297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 268/545] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:23.328837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.448738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163428e6-8522-44fb-85d5-74c83a274f8d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:23.485116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:23.485751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.486058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.486333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.489005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:23.497310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:23.498800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:23.513804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:23.514723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:23.515543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:23.515543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:23.515543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:23.515543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 278/546] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:23.520842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.579510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:23.643747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e786488a-f2f1-4213-b8cd-411823583c45 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:23.692105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:23.692720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.692994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.693259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.693956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:23.694336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:23.696226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:23.712119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 269/547] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:23.717611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.840943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd54c20-1043-4934-b208-0f540a95d6ca None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:23.892327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:23.892972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.893248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.893517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.904894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:23.906432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-61a36e22-d757-48ad-8b87-33260a378780 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:23.920922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:23.922365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:23.924079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 279/548] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:23.927892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:23.986635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0655688-29ee-4e32-a5b1-3d1ee6322b91 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.987180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0655688-29ee-4e32-a5b1-3d1ee6322b91 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.987180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0655688-29ee-4e32-a5b1-3d1ee6322b91 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:23.988740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 280/549] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:23.993883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:23.994187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:23.994460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.036617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:24.038881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:24.095665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:24.107679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511dee4-ff92-4520-89af-fd03f0ee0930 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:24.115509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:24.141140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 281/550] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Oct 28 10:48:24.194715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:24.195335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.195622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.195897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.196645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:24.196956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:24.198443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b998aaaf-7eda-4143-a570-15abdaf7d88f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:24.251067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:24.251402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 270/551] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:23 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:24.401284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86700d87-656a-40b6-ba96-b0090bde54cb None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.401890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86700d87-656a-40b6-ba96-b0090bde54cb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.402471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86700d87-656a-40b6-ba96-b0090bde54cb None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.405408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 271/552] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:24.409689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.410304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.410779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.423080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73aa266e-d3cd-4b9d-a837-eac8c2ec2940 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:24.433705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:24.435405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:24.487401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:24.488036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.488318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.488648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.518082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 272/553] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:48:24.519286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:24.550336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:24.591571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:24.593288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:24.684334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:24.700512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f756ef75-01a6-4f83-b0d9-1e9b15f7090f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:24.716684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 282/554] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 10:48:24.722989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:24.746272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:24.746614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.746904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.747409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.773190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:24.778251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:24.779153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:24.779965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 273/555] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:24.786003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:24.858375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:24.863593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529c21d-98ad-407d-b7df-aba289cce269 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:24.900404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:24.900968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:24.901359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:24.901736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:24.930628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:24.932457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:24.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:24.951573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:24.952418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:24.952418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:24.952418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:24.952418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 283/556] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:24.953671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:24.983407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de332a0b-d1e0-4983-bda1-2c681d11a874 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.019498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.020396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.020853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.021448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.022347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:25.022860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.024446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.033780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 274/557] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:25.039610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.059291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.074977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4494c9c-8903-4a0a-ade8-d437d68ff7af None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.108672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.116892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.117536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.117870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.118103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.130745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:25.136319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:25.137763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:25.138621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:25.142482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 284/558] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:25.143331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.166935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a3d1d9-eb49-4901-8ed2-a990b2a0d869 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.199922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.200633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.200913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.201083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.201697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:25.202018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.203390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.220103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 275/559] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:25.220103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.263233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b279e942-9353-4ebe-96ef-44fb89c1b1bb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.299962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.300598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.300875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.301146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.310645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.314274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.315724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-582b1e83-4743-4916-871f-cf0356306360 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:25.330918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:25.331863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:25.332707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:25.332707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:25.332707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:25.332707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 285/560] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:25.333643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fb817-279b-4113-ae74-8df8f7270430 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.333643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fb817-279b-4113-ae74-8df8f7270430 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.333643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8fb817-279b-4113-ae74-8df8f7270430 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.338144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 286/561] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:25 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:25.338144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8fb817-279b-4113-ae74-8df8f7270430 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.344527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bbf91-6493-478a-acb8-bc793d02299d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.378947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.379546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.379827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.380212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.380332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.380760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:25.381061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.383029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.390788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 276/562] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:25.391565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.448622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8fb817-279b-4113-ae74-8df8f7270430 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.481010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.481579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.481858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.482164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.488580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.490024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.498041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a8265-ab09-4ab8-945b-9023ba7f2643 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7ec66718-7f07-4d55-9135-c1d018c3e0b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:25.501575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:25.502665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:25.503526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 287/563] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:25.504387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.504484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.504731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.522377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:25.523560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:25.532790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.533400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.533681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.533951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.534614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:25.534917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.536278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f570f85-630b-460d-a658-e8e949b69706 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.539065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.548810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 277/564] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:25.549986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eef6b7-32af-419c-b370-3529ea1ba51d None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.550286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eef6b7-32af-419c-b370-3529ea1ba51d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.550562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eef6b7-32af-419c-b370-3529ea1ba51d None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.551743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 278/565] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:25.552426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eef6b7-32af-419c-b370-3529ea1ba51d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.592891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 288/566] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:25.593998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.625397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.665838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eef6b7-32af-419c-b370-3529ea1ba51d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.700544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.701173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.701453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.701725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.703100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0590b7f6-7ac7-4633-8f1a-9f6f25e87b89 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.712057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.713498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:25.728428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:25.729762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:25.730963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:25.730963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:25.730963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:25.730963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 279/567] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:25.730963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.746135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.746951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.747382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.747791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.748665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:25.749284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.751578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86068465-b11b-402f-b44d-ee4c85a018d6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.763542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 289/568] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:25.764524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.764809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.765086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.783543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:25.784627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:25.790994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.843452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5368c2-f818-4e37-9b01-b4034532de65 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:25.856224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 290/569] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:25.857083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.888875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:25.891324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:25.892029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:25.892380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:25.892721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:25.903203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:25.904742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:25.917855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:25.919030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 280/570] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:25.924932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:25.979743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aec4a3-1482-4a8d-b026-0fc18310d648 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.015174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.015722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.016003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.016278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.016919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.017225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.018625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.029695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4e6235-e13c-42df-adcd-dc9e5587ca56 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.030279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 291/571] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.031067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.044594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.071328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.071941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.072222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.072498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.073349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0449fafc849145c8b752a0b028ad2679) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.073660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.075261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.094431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 281/572] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:25 2022] GET /identity/v3/role_assignments?user.id=0449fafc849145c8b752a0b028ad2679&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.095243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.134655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.155345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18788f3-4d5b-4a0f-9b72-07ea7d1963f2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.193756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.195081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.195363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.195625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.207557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:26.213602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:26.214790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:26.215685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:26.216343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9265f4-5f29-4d77-90c9-8e85451c9c99 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.219788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 292/573] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:26.220655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.256959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.257523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.257805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.258114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.258854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.259164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.260604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.281996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 282/574] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.282847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.314772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.334156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788f3d10-5010-4c9f-bad0-134c7fbbe1b6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.370375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.371156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.371444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.371720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.379489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.380918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.392018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b688d5f5-b97b-4492-a7c2-835a92b73c6e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:26.394299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:26.395291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 293/575] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:26.396236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.399330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.429637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.430366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.430649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.431001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.441255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:26.446631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:26.447559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:26.448366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 283/576] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:26.449077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.514773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f669edd2-b921-4f1a-a33d-634c0695f66c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.546696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.550657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.551778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.552355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.552781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.554297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.554767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.556131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.557488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ab4782-e7f7-468f-9071-e62459a2edc8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.565169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 294/577] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.566200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.596458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.597334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.597736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.598059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.609746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.611344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:26.627462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:26.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:26.629299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:26.629299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:26.629299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:26.629299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 284/578] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:26.629519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.643252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.695644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c95a0fb-af6b-4f0b-96f1-45ceb3b277a2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.734738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.735794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.736139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.736476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.737304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.737694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.739223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.766566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 295/579] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.767679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.768165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dda5fe-630b-4a1f-98ef-05feecebfc18 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.823120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.823747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.824030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.824305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.824983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:26.825292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.826702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:26.838174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 285/580] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:26.839580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:26.852447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.935319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db810-e020-4bd4-93d3-1f38b4902d80 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.964361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:26.980763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:26.981584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:26.981950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:26.982344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:26.982449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b296c774-29c6-4fb1-adc1-e71778751ffb None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:26.998037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:26.998037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:27.014503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.015506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:27.016349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:27.016349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.016349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:27.016349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 296/581] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.016563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.035274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.036737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.037637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.038681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.041386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:27.041893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.044831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.066111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 286/582] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:27.067102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.130606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.133876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1494dc76-918f-457d-b34f-f8dced746222 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.168152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9b5ab-49fa-40b7-9834-6a7881541417 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.170952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.171583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.171860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.172128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.183236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.184898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:27.199975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.201071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:27.206741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:27.206741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.206741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:27.206741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 297/583] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.206741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.209163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.210440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.210806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.211147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.221602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.223160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.234731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:27.238038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.239149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:27.240196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:27.240196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:27.240196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:27.243866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 287/584] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.244573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.326691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d31b7f-74f2-466c-b9b2-9fb6772fdda9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.349295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010f8fb9-4b38-4ed7-9b01-64a401509c3e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.362838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.363446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.363722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.363990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.364629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:27.364929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.367024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.369261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.379635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 298/585] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:27.380801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.392176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.392806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.393138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.393414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.401007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.402467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:27.414946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.420547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 288/586] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.421737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.476834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.494041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb0ed7-9713-4cf2-8f08-6ab756d37ec0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.521905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadabc6-a786-4ca6-9122-d19ebdae119c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.530366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.530983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.531265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.531540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.532179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:27.532484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.533876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.550354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 299/587] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:27.550354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.561722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.562444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.562800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.563154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.563872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:27.564323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.565739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.578423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 289/588] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:27.583431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.620097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.657631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ecb0df-ba1c-4beb-b2fb-c915ec1cf3e2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.691620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a92933-033c-4cd4-b66f-3389d6fd0205 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.694804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.695884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.696353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.696776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.705078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.706823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:27.719690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.720827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:27.722024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 300/589] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.727944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.728932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.729900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.730289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.730684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.731398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:27.731771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.733321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.737949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.749681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 290/590] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:27.749681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.844047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:27.846050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3640a782-7d19-4fd2-a129-fe32dad1c4d6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.851467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb15bb-e9fd-4073-9efd-0e36e8ee11c7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:27.883056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.883763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.884103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.884437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.887222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:27.887854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:27.888136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:27.888410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:27.892124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.893611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.896232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:27.898262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:27.906052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.907113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:27.908247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:27.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:27.910541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 291/591] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/users/neutron => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.920251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 301/592] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:27.920251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:27.985127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.021223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5dcb31-4fa9-441d-8b9b-51dd982ba7d7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.021446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f2672-7a86-4a6c-b647-1d51ad799d31 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.054848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.055458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.055744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.056014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.056735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:28.057057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.057330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.058001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.058297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.058497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.058594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.076650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 292/593] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:27 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:28.076951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=6adb6b4eff2545c080d2eb3327b78d0f, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.078013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.078417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.102010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 302/594] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:27 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:28.102010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.103800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.200227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b14562-8f53-4e8c-8b03-a1a40f08d10e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.239252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.239921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.240228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.240417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.241162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:28.241472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.242853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5f0239-5d3d-4ae6-9053-d6518b25e9d0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.243273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.256559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 293/595] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:28.257418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.270391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.295067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.296659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.296947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.297223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.298009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:28.298314 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.299763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a309a16-9848-46dd-8553-5bf37b4ccc52 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.319736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 303/596] 10.222.0.46 () {56 vars in 1139 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:28.321249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b3657-da97-4e0a-a65e-12a07be5df98 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.321623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b3657-da97-4e0a-a65e-12a07be5df98 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.322042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2b3657-da97-4e0a-a65e-12a07be5df98 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.323658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 304/597] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:28.324400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2b3657-da97-4e0a-a65e-12a07be5df98 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.341509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.369559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2ae5d1-65b8-4925-b6d7-7978213b5e0a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.403220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.403845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.404120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.404390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.416776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.418269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:28.433014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:28.434187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:28.435273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:28.435273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:28.435273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:28.435273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 294/598] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:28.435558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.440567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2b3657-da97-4e0a-a65e-12a07be5df98 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.482953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.483525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.483801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.484069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/2297b30fe14d4de8a3f39e247ed59dfb/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.501064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=2297b30fe14d4de8a3f39e247ed59dfb, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.502599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3140c79-fab6-488a-a297-54a73ba23fa3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.516550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 305/599] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:28 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/2297b30fe14d4de8a3f39e247ed59dfb/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:28.517849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.518168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.518447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.544116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:28.545388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:28.553355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aeacaf-4082-49bf-9415-534344f6118a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.575821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.583662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.598023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.598863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.599217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.599584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.616104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.617760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:28.638592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:28.639570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:28.640417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:28.640417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:28.640417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:28.640582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 306/600] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:28.641596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.643091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 295/601] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:28.652476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.769807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dbe2dd-6696-4fdc-a3f1-0b5927b1bc1b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.806861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.807475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.807750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.808015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.808655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:28.808953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.810353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.822243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 307/602] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:28.827636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.829094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a54b8-35f8-496d-b1ec-dc986d2f26f9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:28.846087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.876690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:28.877313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:28.877598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:28.878237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:28.878891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:28.879197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:28.880823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:28.896250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 296/603] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:28.907041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:28.972181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:28.994019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b7e093-1888-488c-9ada-7ee635714af3 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.032920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.033488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.033778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.034079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.044105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.046425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.047867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7834e8-cbcc-4fb4-bd37-fa730af00dd0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:29.062425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:29.064258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 308/604] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:29.065825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.090655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.091568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.091961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.092379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.099846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.101659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:29.114252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:29.115804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:29.118450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 297/605] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:29.119679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.160557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:29.195288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736d9e60-f60d-4866-bca1-72147da21e9e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.231931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:29.236665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08653da2-66b7-4245-b631-8d305a42fd48 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.245127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.245761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.246079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.246361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.260447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:29.269186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:29.270477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:29.271318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:29.273527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 309/606] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:29.274410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.289170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.290557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.291040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.291578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.292502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:29.293009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.294867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.310049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 298/607] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:29.311441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.403241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f55c850-bb27-41d4-a839-c24d5102a26d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.420666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a9b17-040e-453b-8142-e100d94ce2f6 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.440729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.441351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.441631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.441902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.442642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:29.442950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.444937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.446786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:29.461381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 310/608] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:29.462983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.471393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.472274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.472999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.473477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.474390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:29.474897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.476565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.486665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 299/609] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:29.487853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.519439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:29.594546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540223d-573b-4d18-a6de-8498625b6150 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.623335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f07dd14-641e-40e6-a6d1-abec2d6bd977 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.636927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.637600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.637933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.638252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/589508bdaf3c4e159c01e3a43eb69392/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.655756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=589508bdaf3c4e159c01e3a43eb69392, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.658089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.661966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.663463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.663987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.664487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/0287d4acc8f548c482ad40bd61b1081d/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.674459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 311/610] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:29 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/589508bdaf3c4e159c01e3a43eb69392/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:29.675371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:29.682435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=0287d4acc8f548c482ad40bd61b1081d, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.684331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.713585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 300/611] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:29 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/0287d4acc8f548c482ad40bd61b1081d/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:29.752788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:29.826798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eec95fd-f881-48ac-a625-c25f78ecef6b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:29.884660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:29.885718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:29.886259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:29.886778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:29.900846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:29.902790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:29.921775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:29.923223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:29.924103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:29.924103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:29.924103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:29.926323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 312/612] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:29.934528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.012719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:30.059006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0bd809-041a-4229-b4f0-2b8283f6bed0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:30.111986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:30.113020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.113554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.114065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.114922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:30.115471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:30.117117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:30.134843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 301/613] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:30.140715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.289135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:30.298246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadd2f9-f133-49fe-943d-6e5e444f11bb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:30.348461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:30.349098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.349376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.349648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] PATH_INFO: `/v3/users/swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.358204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:30.359653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a17f8119-7214-4008-b63f-182dc1afe0bc None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:30.371102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:30.372094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:30.373093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 313/614] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:30 2022] GET /identity/v3/users/swift => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:30.379775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.462848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786f349-04b8-4408-84c5-fdcbc8d71487 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.463041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786f349-04b8-4408-84c5-fdcbc8d71487 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.463302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786f349-04b8-4408-84c5-fdcbc8d71487 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.464766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 314/615] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:30.469436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.469711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.470016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.494725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:30.495914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:30.526732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be8e747-494b-4a00-bdbf-14a008019876 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:30.585490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:30.590552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:30.591765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.592930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.593246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.594001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:30.594376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:30.595815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:30.616838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 315/616] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 10:48:30.629620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 302/617] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:30 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:30.636213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.735794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:30.753726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3003b-8aaa-4ebb-8bb4-a905899186f2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:30.790427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:30.791476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:30.791968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:30.792453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:30.803162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:30.805554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:30.820741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:30.821728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:30.822704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:30.822704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:30.822704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:30.824405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 316/618] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:30.834215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.952582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:30.988223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679d870-75fa-48b7-a891-2f978f6f1376 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.051843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.052377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.052663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.052937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.053617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:31.053947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.055415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.075416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 303/619] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:31.076144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd35ce-19bc-4ee0-91bb-32cf426c88e2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.080931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.112538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.113542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.114067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.114557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.123259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.129468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:31.136871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:31.137899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:31.138767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:31.146127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 317/620] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:31.148594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.187659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.258282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de46b0-aea2-42ff-b253-c61c4bf856c8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.292781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008e1501-c909-4ac6-9765-9d1577beddff None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.304570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.306175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.306965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.307442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.308323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:31.308836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.310464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860103fc-6e92-4354-a9b9-047d156348f3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.325724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 318/621] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:31.327914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a5c5f-716a-4d09-9197-9d19022807af None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.328438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a5c5f-716a-4d09-9197-9d19022807af None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.328916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a5c5f-716a-4d09-9197-9d19022807af None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.330864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 319/622] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:31.331780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a5c5f-716a-4d09-9197-9d19022807af None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.334846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.335490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.335765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.336029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.344641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.346146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-776c7fb4-c7c5-4ab4-b50e-7a875575db39 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:31.361782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:31.362887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:31.364105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 304/623] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:31.364692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.364692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.364869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.389204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:31.390700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:31.407203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.460765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a5c5f-716a-4d09-9197-9d19022807af None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.478636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 305/624] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:31.480777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.520629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.521729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.523795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.524246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.548877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.550573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.563084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:31.574248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:31.575392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:31.576486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:31.576486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:31.576486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:31.584440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 320/625] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:31.596524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.675328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a2323c-2fe3-4b88-8570-927bb9eff444 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.742528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.743459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.744040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.744459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.745300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:31.745765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.747426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.759963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 306/626] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:31.766087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.777909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d6cef-6c7e-4736-ad7c-68dd2d08de2d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.801915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.827683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.828432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.828664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.828937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.829639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:31.829941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.831453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.849575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 321/627] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:31.849575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.893247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:31.896885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40152e3-028e-472c-833d-91cb57a0fa61 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:31.935419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:31.936128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:31.936409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:31.936679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/0449fafc849145c8b752a0b028ad2679/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:31.952242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=0449fafc849145c8b752a0b028ad2679, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:31.953697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:31.971184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 307/628] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:31 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/0449fafc849145c8b752a0b028ad2679/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:31.972130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:31.983124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea82ba-5a97-4c69-a91c-e88b22c0911b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.057563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.058656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.059186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.059663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.087325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:32.094480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:32.095567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:32.096422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:32.100106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 322/629] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:32.106628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.163259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d888e31-6be0-44f5-af41-3a6885b859f1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.188328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.205633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.206359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.206634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.206900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.216629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.218135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:32.235235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:32.236249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:32.237323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 308/630] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:32.244561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.258764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.296095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8bb2b0-f989-4d15-a896-4c2b9685339b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.347362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.348160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.348441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.348720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.349405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:32.349709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.351748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62305507-3c55-45d6-99ac-bcf3c123d90a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.362011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 323/631] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:32.363736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1af95-4509-4b2c-a956-cfe3aa1929f9 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1af95-4509-4b2c-a956-cfe3aa1929f9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.364297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb1af95-4509-4b2c-a956-cfe3aa1929f9 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.370450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 324/632] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:32.371358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb1af95-4509-4b2c-a956-cfe3aa1929f9 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.385484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3730167-6a44-45f3-aaa1-40ca34b6c232 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.422758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.423439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.423722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.423962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.424707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:32.425015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.426636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557749c1-9abe-4c11-86fb-4b1ea5371033 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.451150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 309/633] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:32.451150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.451150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.451150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.467903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:32.469161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:32.492943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.501501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb1af95-4509-4b2c-a956-cfe3aa1929f9 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.537693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.538938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.539217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.539479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.540238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.549058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 310/634] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:32.549878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.558578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.560878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5628b72e-66c0-4b92-8991-8310dd89d5d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:32.581855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:32.583211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:32.584316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:32.584316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:32.584316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:32.586653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 325/635] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:32.588049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876d080-ff2d-4906-868a-a550392fc145 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.588671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876d080-ff2d-4906-868a-a550392fc145 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.589097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876d080-ff2d-4906-868a-a550392fc145 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.591338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 326/636] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:32.592210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2876d080-ff2d-4906-868a-a550392fc145 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.665470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cecefb-fef2-4844-88c2-ec4122557fc6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.702987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.703858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.704313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.704736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.727166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2876d080-ff2d-4906-868a-a550392fc145 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:32.770565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.773543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.775195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.775794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:32.776580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.777038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.777647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.779311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:32.779777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:32.781284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28fd17d-8c4f-4614-892d-58650fd32059 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:32.797725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 327/637] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:32.799230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:32.799652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:32.800074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:32.815907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:32.823093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:32.824320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:32.886399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 311/638] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:32.887275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:32.910974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 328/639] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:32.911821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.005633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.011758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602c1192-fb76-49a0-8ab5-4a2b9f8bf438 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.033742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188803c-3e2c-4b77-a9d6-f7d7237562d5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.048518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.049025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.049311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.049583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.057149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.058516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:33.073817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.075101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 312/640] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.076274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.081649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.082610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.083057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.083488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.088324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.098314 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:33.103542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.104888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:33.105713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 329/641] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.106427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.209510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9effe8c5-1e2f-472c-8b41-d8f84b7620aa None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.215636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dc1c1a-251c-4b25-940c-a1716a6005db None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.246399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.247337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.247693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.248026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.253717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.254362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.254646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.254922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.256243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:33.257042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.258955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.259436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.260863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:33.276113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.277113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:33.278024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:33.278024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:33.278024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:33.278024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 313/642] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.278024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.283881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 330/643] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:33.284711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.291612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.342429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.390712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8edc55-81bf-4eac-84eb-cc9318352d7f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.393751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98898f74-f649-4e8a-b904-9b7ab55ad2f8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.427384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.428116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.428561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.428840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.430272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.430888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.431179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.431446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.432117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:33.432422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.433796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.440974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:33.446125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.447053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:33.448033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:33.448696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 314/644] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:33.448696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.450242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 331/645] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.451429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.521739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.554721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca997d06-6580-4a15-b844-e11707105561 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.557437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fd51ab-7244-4d6f-bfa9-2e101f956edb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.572799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.588129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.588737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.589011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.589282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.589923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:33.590261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.591627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.595857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.596616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.597041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.597451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.606080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 315/646] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:33.606080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.673238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.674848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.708133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747480f0-a390-452c-8cf3-3f535fdae37b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.739969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.746325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.746963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.747243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.747516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.758487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.759967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:33.775509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.776604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:33.777624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:33.777624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:33.777624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:33.777624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 316/647] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.778780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.785400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 332/648] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:33.786265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.802676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:33.888359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094bc2f-0e09-4aa6-baa4-04d2c83daa91 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.893048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755243-4636-43f7-8221-e246be58b649 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:33.925653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.926308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.926646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.926893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.927512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:33.927813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.929250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.930799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:33.931757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:33.932039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:33.932308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:33.937535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 317/649] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:33.938176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.940800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:33.942453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:33.954466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:33.955416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 333/650] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:33.956287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:33.992525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.039509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.053655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842822a-99f9-440d-a9d2-6586d01c0ca6 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.065301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c6338-a737-44dd-82d0-6bd9a9909ff0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.090076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.090684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.091119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.091393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.092666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.092978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.094850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.102834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.103989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.104274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.104806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.113469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 318/651] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.113469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.116872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.119218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:34.134649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:34.135576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:34.136432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:34.136432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:34.136432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:34.136432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 334/652] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:34.137522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.219941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170c41c3-0c5e-4353-9979-9ffcc19659d1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.222641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.244044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b69ff5-6555-4fc8-bb5a-55e4faa559f0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.265589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.266544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.266973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.267385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.278856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.279259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.281400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.288828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.289577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.289866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.290226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.290973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.291341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.292822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:34.296830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:34.297752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:34.298858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:34.298858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:34.298858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:34.298858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 319/653] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:34.299117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.310274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 335/654] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.311232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.407113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aa8ef4-b2d9-4de7-8b38-e7ba11f10eb2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.408573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd63e6-6d6f-4131-b161-27ce7db30ac7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.441787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.442296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.442534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.442698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.442796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.442979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.443077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.443281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.443933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.444246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.445614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.451106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.452529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.456683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 336/655] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.457583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:34.463920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:34.470470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 320/656] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/users/glance => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:34.471475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.484497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.508336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.561084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aad833f-4480-4e2a-98a0-299bb5c72cb9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.564232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba0c98-0e4f-430f-8ac3-ffa72e022b60 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.594169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.594780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.595056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.595326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.595958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.596257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.597315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.597610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.597906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.598207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.598476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.599189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.599490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.600929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3ec384-8c81-4b0e-aa40-d47abf9a8a20 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.606794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 321/657] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.607902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3dd93-2563-4ab1-823d-b47c37717eef None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.608178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3dd93-2563-4ab1-823d-b47c37717eef None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.608445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef3dd93-2563-4ab1-823d-b47c37717eef None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.613736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 337/658] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.613866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 322/659] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:34.613962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.614189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef3dd93-2563-4ab1-823d-b47c37717eef None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.709702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.734091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef3dd93-2563-4ab1-823d-b47c37717eef None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.736367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c60ce4-bf62-4058-9a4b-61281a433c92 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.745217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.771841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.773024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.773302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.773572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.774361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:34.774671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.775741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.776115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.776378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.776652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.776917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.784677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.786162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.798881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 323/660] 10.222.0.46 () {56 vars in 1137 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:34.798881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c7594526-666a-4dc6-9cba-446ac63d636b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:34.799920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:34.800747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:34.801594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 338/661] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:34.802521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.802521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.802787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.823688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:34.824898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:34.902967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 339/662] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:34.903854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:34.916077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e68d8d-7ac1-46cf-a082-3129b1b122de None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:34.954729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:34.955730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:34.956206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:34.956682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:34.967910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:34.969688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:34.972768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:34.987981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:34.988928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 10:48:34.989931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:34.996768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 324/663] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/users/neutron => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:34.996768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.015163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.024007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6063597e-120b-4f49-8aeb-33f9409f794a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.071593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.072743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.073168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.073581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.107408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884c256-49a6-4139-a3ce-082e556512ad None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.145702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.146501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.146925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.147322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.148265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:35.148654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.150413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.163949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.164134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 325/664] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:35.164798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.165934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.234291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.272911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.273161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 340/665] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:35.275240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.291673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a565ec-a09d-4af3-9acc-63929904a19a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.329033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.329419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.329691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.329968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.330718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:35.331029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.332617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.350718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 326/666] 10.222.0.46 () {56 vars in 1139 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:35.351543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.380554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf2aabd-f559-4bdc-80a5-6c45f03712c8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.418670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.419337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.419539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.419810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.420530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=2297b30fe14d4de8a3f39e247ed59dfb) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:35.420836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.422276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.438013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 341/667] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/role_assignments?user.id=2297b30fe14d4de8a3f39e247ed59dfb&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:35.438013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.458443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca47dadf-1a32-4431-975b-f1ec61070acb None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.494704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.499685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.500441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.501232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.501595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.515298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.516359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:35.521656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:35.522811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:35.523616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:35.528752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 327/668] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:35.529802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.575568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6046fc-0c2a-4cbc-a9b1-d5536efd7054 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.622294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.624266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.624758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.625229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.644372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.644547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f30a23-ec8c-450f-af7d-9f80272d09f3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.644629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:35.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:35.665191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:35.666069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:35.666069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:35.666069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:35.671734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 342/669] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:35.671734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.684766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.685384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.685665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.685936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.735561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.764123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.765568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.797784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c2943a-6ef1-46af-b680-cec78fb4fcea None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:35.812047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.840861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:35.842093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:35.842842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:35.843765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:35.844934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:35.845694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:35.848000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:35.859565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 343/670] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:35.861220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.892523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 328/671] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:35.893396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:35.980548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:35.991969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699b6c90-0234-439b-8633-5c50f28ce925 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.014653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9989c01e-208d-4ba4-bd36-05a62bbb04e3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.028088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.028701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.028978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.029510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.030100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.030408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.031831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.044886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 344/672] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.045770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.062102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.063466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.064159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.064619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.078731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.081170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.085917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:36.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:36.102423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:36.103277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:36.103277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:36.103277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:36.104816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 329/673] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:36.105814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.163964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aed8f-8fd3-4a4f-a052-c9997152e2b1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.200322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.201027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.201307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.201631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.212857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.214321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:36.230422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:36.231431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:36.233068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:36.233068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:36.233068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:36.233068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 345/674] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:36.233068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.236653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006c585-6a7f-4a2f-96c8-11250f00bb47 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.280776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.293661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.294839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.295298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.295745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.306092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.308461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:36.323027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:36.323938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:36.324814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:36.327050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 330/675] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:36.328023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.337360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.355143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9a09e9-ac65-4487-8fc0-da9ba51ee8c1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.395328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.396589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.397657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.398218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.399256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.399722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.401867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.418829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 346/676] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.420421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.458762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d388ce91-e426-4e72-99c4-4ee1b6fc39e8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.497192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.497947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.498327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.498690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.499487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.499870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.501333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.514313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 331/677] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.515351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.549254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e438-e5f9-4b12-9030-6b793896a832 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.565722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.591927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.592686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.593055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.593421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.594356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.595506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.596506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.614073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 347/678] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.615058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.625771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.633158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092a1d54-eac6-4478-a7c4-c0cd4fe64316 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.675036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.676221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.676651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.677319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.686955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.689329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:36.707693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:36.708850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:36.710037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:36.712251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 332/679] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/users/swift => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:36.713226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.731496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03294d5a-1de2-42b7-979b-212c39e06a52 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.773280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.774088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.774458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.774822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.782515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.784122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:36.797277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:36.798237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:36.799353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 348/680] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:36.800820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.858374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc544764-5a71-4d02-b889-69ede9e9101d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.868424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.871139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:36.895872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.896467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.896745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.897036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.897755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=589508bdaf3c4e159c01e3a43eb69392) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.898081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.899551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.914020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 333/681] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/role_assignments?user.id=589508bdaf3c4e159c01e3a43eb69392&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.914020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:36.921334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec2b9e2-3903-449f-a6b1-d8c1f773156d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:36.964582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:36.965203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.965480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.965834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.966609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:36.966913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:36.969086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf84d02-a09b-4f45-838d-bd2296eeeca4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:36.994196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 349/682] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:36.995395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc403ee8-38f7-4c9e-9e7a-14ed67583b07 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:36.995715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc403ee8-38f7-4c9e-9e7a-14ed67583b07 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:36.996007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc403ee8-38f7-4c9e-9e7a-14ed67583b07 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:36.997446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 350/683] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:37.003049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc403ee8-38f7-4c9e-9e7a-14ed67583b07 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.041004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d05a7b-2af7-4f4a-ac0b-56e052393087 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.082831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.083634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.083984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.085118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.086174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:37.086515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.088026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38989517-562f-472c-a9cc-d7a12ade18ea None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.102786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 334/684] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:37.104197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.104478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.105066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.123720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:37.125000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:37.126437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc403ee8-38f7-4c9e-9e7a-14ed67583b07 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.155880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.158385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.173227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.173842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.174166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.174439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.207768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 335/685] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:37.208717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.262552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.264557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.335743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b0d42d-92dc-46e4-8a9b-2695ab1b2618 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.383171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.384083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.385077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.385506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.386574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0287d4acc8f548c482ad40bd61b1081d) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:37.387132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.389185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.396802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 336/686] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/role_assignments?user.id=0287d4acc8f548c482ad40bd61b1081d&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:37.408458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 351/687] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:37.414301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.444585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.534764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31fbef-7214-4f32-89e3-c1d7cdb57330 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.552902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.573034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.573799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.574104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.574374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.586600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.588092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.597449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:37.606425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:37.607445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:37.608324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:37.608324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:37.608324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:37.608324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 337/688] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:37.613474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.711671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67250be8-e111-47d0-8d8b-882824df5d08 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.740015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.759318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.759964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.760238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.760454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50f463-ff73-4417-afa7-37cbd6382135 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.760583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.773483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:37.779573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:37.780504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:37.781334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 352/689] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:37.795005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.798513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.799300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.799576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.799847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.800492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:37.800797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.802362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.815722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 338/690] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:37.822530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:37.911820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:37.931386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b617bae3-9453-4a08-b07a-c465f898b35b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.967346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1504fda-45d8-44b7-af55-3b843754750a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:37.971135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:37.971464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.971736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.972002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.972665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:37.972962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:37.974364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5ebeb0-645e-471c-a589-41adaef1db5f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:37.990026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 353/691] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:37.990026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a95d1d-996b-4779-89b9-87466fe749ce None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:37.990026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a95d1d-996b-4779-89b9-87466fe749ce None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:37.990026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a95d1d-996b-4779-89b9-87466fe749ce None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:37.990026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 354/692] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:37.991842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a95d1d-996b-4779-89b9-87466fe749ce None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.015771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.016648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.017071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.017567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.025310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.027207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.031998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-01ccb6d1-7e11-4cbf-a81a-907baf83cad5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:38.047141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:38.048144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:38.049019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:38.051850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 339/693] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:38.054302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.054796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.055260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.086971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:38.089358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:38.120491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a95d1d-996b-4779-89b9-87466fe749ce None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.161078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.161712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.162058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.162294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.173356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.174820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.182405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.182982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 340/694] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:38.185328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:38.191456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:38.192456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:38.194611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:38.194611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:38.194611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:38.194611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 355/695] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:38.198878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.311956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4a65a4-3876-4809-8957-3d1d5ef5640f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.347364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.348431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.348907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.349414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.350839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:38.351353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.352923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.364045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e68091f-f5e6-4a4c-80f7-e49a0f2afbc9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.367422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 356/696] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:38.375428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.437862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.443172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.463653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.464167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.464442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.464714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.465408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:38.465719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.467157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.488824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d8efc9-202a-434a-b623-73ae9c5f0e41 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.496391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 341/697] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:38.502156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.527612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.528258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.528715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.529004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.537453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.539477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:38.553162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:38.555494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:38.557180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:38.561124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 357/698] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:38.561124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.625759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf95cdb-4b83-4290-92da-e6b635c2a87f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.671070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.672093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.672376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.672650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.673666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0449fafc849145c8b752a0b028ad2679) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:38.673996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.676103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.678928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b67702-1aeb-426e-a640-64ec2cbabc1d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.683921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 342/699] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/role_assignments?user.id=0449fafc849145c8b752a0b028ad2679&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:38.684763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.701822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.727256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.728563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.729426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.730343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.731248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.732920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:38.745800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:38.746927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:38.747795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:38.750809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 358/700] 10.222.0.46 () {56 vars in 1129 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:38.760681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.777222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:38.873274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f5764-3540-4dc1-8dbc-811fc6d84cb1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.923789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55de15fe-ebd6-4ee4-9b9a-620035a42638 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:38.931353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.932805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.933284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.933750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.934952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:38.935459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.937724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.943295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 359/701] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:38.948609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:38.962897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:38.963592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:38.963814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:38.964061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:38.964784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:38.965108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:38.966511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:38.987460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 343/702] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:38.993228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:39.005521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:39.074745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6cacc-8a46-410d-8682-a6240fa0560a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:39.112684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc48b66-f546-4dfa-8233-5bbe28bcfadd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:39.115951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:39.116570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.116845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.117113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.117745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:39.118075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:39.119524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:39.125756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 360/703] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:39.126878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:39.171613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:39.172241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.172516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.172781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.272918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:39.274713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78128976-98bc-4803-a22e-dbaf447fae58 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:39.383960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 344/704] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:39.389569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:39.537513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8f027b-06be-4a31-855e-347779b3e07a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:39.554741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:39.556859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f70c49-4f50-4743-bf13-7a893652e59b None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.557141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f70c49-4f50-4743-bf13-7a893652e59b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.557410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f70c49-4f50-4743-bf13-7a893652e59b None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.558847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 345/705] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:39.565224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.565635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.566004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.589032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:39.589911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.590408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.590818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.602599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:39.604426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:39.607047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:39.608384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:39.615018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:39.621917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:39.622881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:39.623741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:39.623741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:39.623741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:39.623741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 361/706] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:39.630347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:39.744188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 346/707] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:39.807534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42f39ca-3690-47b3-a86d-b238d9f3a777 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:39.849409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:39.850429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:39.850942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:39.851421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:39.852284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:39.852778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:39.854360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:39.872395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 362/708] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:39.878078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.013288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1f06b0-0f7f-45cc-8dcc-ecd0953ef39b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.044927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.053804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.054498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.054781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.055060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.071026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:40.071180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.072504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.079295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:40.097608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:40.098596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:40.099453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:40.107096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 347/709] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:40.107096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.224825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80130b4-aa08-44c4-a9fd-4fc3a03fe264 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.288502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.289420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.289741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.290462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.299936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca3918-611a-491e-b0c3-c857beaeb261 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.304506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.305943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-9cdb44d6-1498-4d65-b869-afe20e50119f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:40.325269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:40.326339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:40.327207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:40.327207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:40.327207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:40.330128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 363/710] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:40.331448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa4ed9-abe5-46aa-a64e-f481b49f6fb2 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.332444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa4ed9-abe5-46aa-a64e-f481b49f6fb2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.332715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa4ed9-abe5-46aa-a64e-f481b49f6fb2 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.334023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 364/711] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:40.335580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfa4ed9-abe5-46aa-a64e-f481b49f6fb2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.347976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.348946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.349467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.350061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.351681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:40.352192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.354007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2579d2-7861-4ae4-925f-a70038c0605b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.368279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 348/712] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:40.369999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.370563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.371115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.391940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:40.393601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:40.425081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:40.451456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfa4ed9-abe5-46aa-a64e-f481b49f6fb2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.458649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:40.478250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 349/713] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:40.479876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.495596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.496217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.496494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.496764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.497659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:40.497780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.499070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.526483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 365/714] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:40.533616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.596807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d19ff1-506b-4c1b-8186-ca94b40c396b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.634860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.635835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.636322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.636799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.637748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:40.638284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.639927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.646581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 350/715] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:40 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:40.703378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b491539-5c9f-41d2-a200-597f0c4dbeeb None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:40.758187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:40.758820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:40.759095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:40.759361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:40.760014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:40.761438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:40.773142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:40.818239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 366/716] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:48:40 2022] POST /identity/v3/users => generated 254 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:48:40.906710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:40.976229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:41.026844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c57e957-73c8-42c6-aa35-2624649b1a05 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:41.064054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:41.064707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.064989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.065261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] PATH_INFO: `/v3/services/network` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.065851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:41.067268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:41.079767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:41.080809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:41.081643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 351/717] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:40 2022] GET /identity/v3/services/network => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:41.087166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:41.216808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd69ba6-1b63-4203-846e-334acd00a6f2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:41.255665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:41.256303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.256591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.256869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.257515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] RBAC: Adding query filter params (name=network) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:41.257813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:41.259310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3045dc-5ab2-4000-994d-34394fd41bdd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:41.264240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 367/718] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:41 2022] GET /identity/v3/services?name=network => generated 124 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:41.269824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:41.377174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37074fce-9241-4a75-a6cc-029d8f310074 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:41.399209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:41.420932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:41.421901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.422419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.422909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.423759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:41.424253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:41.425835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8fce71-261b-4463-acac-f358b52e99c4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:41.438744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 352/719] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:41 2022] GET /identity/v3/services?type=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:41.511595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbc099-5bce-4bdd-a567-31c300fcee82 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.511781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbc099-5bce-4bdd-a567-31c300fcee82 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.512065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bbc099-5bce-4bdd-a567-31c300fcee82 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.513612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 368/720] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:41.519675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdf2683-cbaf-4748-bf3e-00201c69fbbd None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.520272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdf2683-cbaf-4748-bf3e-00201c69fbbd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.520738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdf2683-cbaf-4748-bf3e-00201c69fbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.541043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5fdf2683-cbaf-4748-bf3e-00201c69fbbd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:41.542669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5fdf2683-cbaf-4748-bf3e-00201c69fbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:41.632816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 353/721] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:41.744653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4074-4976-48ef-944d-d6b237c82db5 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.745154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4074-4976-48ef-944d-d6b237c82db5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.745259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050e4074-4976-48ef-944d-d6b237c82db5 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.746827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 369/722] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:41.752121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:41.752409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:41.752683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:41.772798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:41.773924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:41.850389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 354/723] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:41.959838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050e4074-4976-48ef-944d-d6b237c82db5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.036694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:42.097662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050e4074-4976-48ef-944d-d6b237c82db5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:42.146765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:42.147442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:42.147724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:42.148001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:42.164070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:42.170480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:42.171390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:42.172213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 370/724] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:42.177372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.189814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.258006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:42.355611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75dd8ef-05b1-462b-94b1-7e3cb38bc236 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:42.366931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0b8991-333e-47ab-b367-2da93c8e2e65 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:42.390335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:42.421119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:42.421741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:42.422041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:42.422318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:42.422999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:42.423304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:42.424675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:42.430735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:42.431354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:42.431636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:42.431899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:42.432562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:42.433928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:42.445806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 355/725] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:42.452998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.469285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 371/726] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:48:42 2022] POST /identity/v3/services => generated 251 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:42.592815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccc0d0a-8383-435a-8768-47e4d3151ad7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:42.595302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:42.636485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:42.637064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:42.637345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:42.637622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:42.654050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:42.655522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:42.677808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:42.679438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:42.680348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:42.680348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:42.680348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:42.680348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 356/727] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:42.685816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.818198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b85169c-b0b3-4914-98f1-04e6dcc6d5d7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:42.855428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:42.856050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:42.856328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:42.856599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:42.857246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:42.857549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:42.858945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:42.871926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 372/728] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:42.877491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:42.998452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254813f1-cbeb-4ae2-9805-8264fbfe900a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:43.035983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:43.036627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.036910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.037191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.045013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:43.046503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-50dba3d2-59dc-4128-8cc9-ddb5ce55d569 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:43.057869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:43.058850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:43.059789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 357/729] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:42 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:43.064858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:43.101129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:43.195030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b16666-0730-437a-bc8f-39a5857164af None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:43.239810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:43.240431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.241259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.241538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.242306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:43.242611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:43.244043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:43.276048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 373/730] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:43 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:43.280497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3447090-6bf9-4660-a628-01463cd4015e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.280947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3447090-6bf9-4660-a628-01463cd4015e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.281025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3447090-6bf9-4660-a628-01463cd4015e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.282378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 358/731] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:43.284882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:43.296538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.296812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.297088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.331956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:43.333157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:43.350328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:43.403219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:43.417489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50dbd1-901a-4bee-94e6-796e4416612b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:43.428823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 359/732] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:43 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:43.482688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:43.483373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.483659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.483938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.576024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:43.577492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:43.697403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:43.705829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:43.712873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 374/733] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:43 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 10:48:43.718013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:43.898585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94c1203-31d1-49b7-b650-84c3b5f31c21 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:43.920429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3acca1-e7ff-4b49-a5b6-d030e58d65e9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:43.940529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:43.941171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.941443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.941709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.962935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:43.964389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:43.980247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:43.980883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.981157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.981424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-bb3f7cb7-a2de-40f1-a70c-5f8c08236bea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:43.986825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:43.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:43.988627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:43.988627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:43.988627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:43.988627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 375/734] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:43.989958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac312-7d35-4871-b36c-0a7458dc6caf None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:43.990257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac312-7d35-4871-b36c-0a7458dc6caf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:43.992331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8ac312-7d35-4871-b36c-0a7458dc6caf None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:43.992331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 376/735] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:43.992622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:43.997075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8ac312-7d35-4871-b36c-0a7458dc6caf None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-73b30ff4-bbfb-41ef-a95b-24eb2efae1c2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:43.997473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:43.998408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:43.999308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 360/736] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:44.001222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.001222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.001222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.013890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.028011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:44.029738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:44.055960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.113845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 361/737] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:43 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:44.114861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.132695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8ac312-7d35-4871-b36c-0a7458dc6caf None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.193486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.194517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.195987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.196459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.197676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:44.198209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.201167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.228827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 377/738] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:44.234531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.343374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c937993-72d8-45de-8d9a-320192eeeab2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.371948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a88e908-1006-4178-9a79-7dc90a721d3d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.374485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.377624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.385506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.386325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.386680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.387104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.387841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:44.388215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.389828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff8e30b-a26d-4c69-a735-49a328ce5fd1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.404453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 362/739] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:44.405968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f5e4-bf12-423d-9465-5c6eba1321eb None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.406301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f5e4-bf12-423d-9465-5c6eba1321eb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.406580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5f5e4-bf12-423d-9465-5c6eba1321eb None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.407812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 363/740] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:44.408505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5f5e4-bf12-423d-9465-5c6eba1321eb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.413389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.414056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.414336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.414606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.421553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.423040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:44.435008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:44.436056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 378/741] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:44.436929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.525630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5f5e4-bf12-423d-9465-5c6eba1321eb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.546372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0be08b-d2ab-4bcc-bf04-4659fee1fa08 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.559268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.559893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.560168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.560713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.571913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-33fb0fa5-9b45-42d7-b380-128390e40077 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:44.576706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:44.577791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:44.578647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 364/742] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:44.579413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.579692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.579960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.587136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21797961-daba-411a-a814-483085d7da06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.588034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21797961-daba-411a-a814-483085d7da06 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.588509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21797961-daba-411a-a814-483085d7da06 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.588995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21797961-daba-411a-a814-483085d7da06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.597862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:44.599072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:44.600644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21797961-daba-411a-a814-483085d7da06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.602256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21797961-daba-411a-a814-483085d7da06 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-21797961-daba-411a-a814-483085d7da06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:44.618196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:44.619137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:44.624708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:44.624708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:44.624708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:44.624708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 379/743] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:44.624708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21797961-daba-411a-a814-483085d7da06 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.642965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.654683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.674920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 365/744] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:44 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 10:48:44.676028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.758229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21797961-daba-411a-a814-483085d7da06 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.797454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.798451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.798906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.799352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe749f5b-f3ef-42de-b4f4-e59b224886a7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.799572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.800443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:44.801018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.802732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.818280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 380/745] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:44.820114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.851091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.851792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.852134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.852733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.853859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:44.854269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.856473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.868323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 366/746] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:44.869222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:44.911797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.932187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:44.944441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466430b1-d431-4f4e-aeb1-586f6cd187f4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.983661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:44.983945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:44.984213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:44.984486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:44.985124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:44.985428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:44.986822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:44.996902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fdddb-9036-4c1f-80e5-f48ab3da1835 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:44.999213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 381/747] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:45.000414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.034600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.035210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.035492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.035765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.036487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0449fafc849145c8b752a0b028ad2679) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:45.036797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.038813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.050087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 367/748] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/role_assignments?user.id=0449fafc849145c8b752a0b028ad2679&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:45.050087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.114778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d53c3-cc5c-45c2-8767-56269b199968 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.157195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.157848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.158155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.158425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.161953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.173453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.174956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:45.191273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:45.192299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:45.193176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:45.193176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:45.193176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:45.193176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 382/749] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:45.193176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.203642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b24e769-d667-4a81-be00-365dcad8f0f7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.218648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.250086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.253243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.253243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.253243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.253243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.255984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.277423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 368/750] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:48:45 2022] POST /identity/v3/services => generated 240 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:45.278104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.358876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac19a28-11e2-418d-8c59-7d0a59755783 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.411962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.413233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.413524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.413799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.415460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3ce0bd-d9e4-4751-b339-1e4a88c826ca None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.422411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.424158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-0809af21-4348-4b9b-88a9-e9881c99e575 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:45.438137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:45.439253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:45.440666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 383/751] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:45.441947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688ce6d-a9ab-41d6-9795-01f881632ad6 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.445037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688ce6d-a9ab-41d6-9795-01f881632ad6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.445037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3688ce6d-a9ab-41d6-9795-01f881632ad6 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.448117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 384/752] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:45.449026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688ce6d-a9ab-41d6-9795-01f881632ad6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.458656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.460576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.460855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.461118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.461782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:45.462619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.464492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa66748-49a4-4d9e-aa24-b943951c327a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.482327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 369/753] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:45.483608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.483960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.484240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.503294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:45.504524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:45.511088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.539730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.580203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3688ce6d-a9ab-41d6-9795-01f881632ad6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.592502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 370/754] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:45.593679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.620728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.621582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.622048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.622473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.623657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:45.624581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.626581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.648484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 385/755] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:45.654486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.755697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead5ce7f-4ea9-41e9-936f-ffeaa940b8da None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.769754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b96e8-04c2-4628-9b97-3c797c342bc7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:45.771514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.810252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.811136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.811568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.812453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.824680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.826356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.830581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:45.831229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:45.831506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:45.831771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:45.844411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:45.845600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:45.846722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:45.846722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:45.846722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:45.848515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 386/756] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:45.853096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:45.853096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:45.854284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:45.873481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:45.874485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:45.875500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 371/757] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:45.880542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:45.910571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:45.989189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069c353-acb7-4420-a301-78a21138a2f9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.028133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894a7c2-825b-47a8-9bd0-efb3b54e199e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.030815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.031608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.031965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.032331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.033059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:46.033433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.034959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.052707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 387/758] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:46.053842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.069411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.070177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.070695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.070980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.072184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.072602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.074383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:46.088217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:46.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:46.090073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:46.090804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 372/759] 10.222.0.46 () {56 vars in 1129 bytes} [Fri Oct 28 10:48:45 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:46.091719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.168059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71957036-3bab-4fd7-af4b-ba0e82aa4f0b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.209342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.213375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.214763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.215259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.215970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.216957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:46.217460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.219116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.228329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb157ec-d720-414c-b589-53f8b74ae069 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.239529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 388/760] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:46.241103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.276962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08581928-1437-421f-8de9-3531d9facadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.277884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08581928-1437-421f-8de9-3531d9facadb None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.278387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08581928-1437-421f-8de9-3531d9facadb None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.278838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08581928-1437-421f-8de9-3531d9facadb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.286162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08581928-1437-421f-8de9-3531d9facadb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.288119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08581928-1437-421f-8de9-3531d9facadb None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-08581928-1437-421f-8de9-3531d9facadb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:46.301265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:46.302251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:46.303666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:46.305821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 373/761] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:46.306852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08581928-1437-421f-8de9-3531d9facadb None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.336046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.368943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efeb75c-7dd9-406c-96ef-a57273a05193 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.411045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.412432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.412930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.413638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.414526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:46.415171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.417306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.423823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 389/762] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:46.424350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08581928-1437-421f-8de9-3531d9facadb None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.425397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.461732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.462392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.462724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.462996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.478530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.536861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.538887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.552706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1363-b11a-43aa-8433-f66c327438fc None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.597320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.598784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.599245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.599715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.601567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:46.602198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.604067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.607774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.626019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 390/763] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:46.627029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.654700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 374/764] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:48:46.655795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.721702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:46.746742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c8217e-e686-4d07-9692-d6cbe965de48 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.769636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c4d5f-8957-4e66-a451-6947502a9df9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:46.788940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.789744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.790188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.790605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.791398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:46.792254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.794027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.800830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 391/765] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:46.802100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.807525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:46.808137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:46.808537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:46.808980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:46.811009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=6adb6b4eff2545c080d2eb3327b78d0f, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:46.812596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:46.834064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 375/766] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:46 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:48:46.835349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:46.907444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.025027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ec9fbf-8692-4dac-a17d-1711d892cbe0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.086292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.087776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.088495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.089395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.110432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.112209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a5b24-b8ff-416d-8f0c-bc96303da6c0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.113703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.116320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:47.140201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:47.142009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:47.143749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:47.143749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:47.143749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:47.147747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 392/767] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 346 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:47.161193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:47.185132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.185708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.185941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.186246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.187021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] RBAC: Adding query filter params (interface=public, service_id=4eb6087752c84f888e5acf53392e9afe, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:47.187335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.189082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.197456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 376/768] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:48:46 2022] GET /identity/v3/endpoints?service_id=4eb6087752c84f888e5acf53392e9afe&interface=public®ion_id=RegionOne => generated 194 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:47.206002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:47.313239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75141ed-1442-4a04-afa3-bd64e5c992cf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.340195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.379779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.380394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.380670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.380942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.382157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:47.382461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.383871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.391210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aad0dd-ab82-4589-a9f1-f2ed1a5a5dbd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.415280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 393/769] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:47.420516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:47.472771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.473466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.473700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.473989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.474599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.476509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cb61fa-6e1e-46ea-a1d2-d7e415d7e9ff None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.490120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 377/770] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:48:47 2022] GET /identity/v3/services => generated 772 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:47.536442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87bffb1-b503-4df2-a9cf-b9d44f0d6cfd None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.537105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87bffb1-b503-4df2-a9cf-b9d44f0d6cfd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.538219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87bffb1-b503-4df2-a9cf-b9d44f0d6cfd None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.539908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 378/771] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:47.546204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90149c-b4d7-4be9-b237-6960308ae065 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.546831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90149c-b4d7-4be9-b237-6960308ae065 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.547445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90149c-b4d7-4be9-b237-6960308ae065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.569946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-aa90149c-b4d7-4be9-b237-6960308ae065 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:47.571299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aa90149c-b4d7-4be9-b237-6960308ae065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:47.571703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c980fe-2cf4-4799-a61e-a6ab0bf319de None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.603825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.615749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.617653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.618119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.618567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.633724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.636558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:47.662196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:47.663182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:47.664038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:47.664977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 394/772] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:47.671345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:47.689475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 379/773] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:48:47.721016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.726425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543fa99-79fc-4a4d-bf2a-2a2fb8146223 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.727014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543fa99-79fc-4a4d-bf2a-2a2fb8146223 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.727490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543fa99-79fc-4a4d-bf2a-2a2fb8146223 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.730706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 380/774] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:47.734996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.735652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.736178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.755951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:47.758067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:47.797494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1250f10e-f7ad-4458-9f7d-c49b32c885a4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:47.835614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:47.836232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:47.836507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:47.836818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:47.837516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:47.837810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:47.845996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:47.846135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 381/775] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:48:47.858443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 395/776] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:47.864384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:47.968341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:47.981780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f279087-3fc9-4da2-ac64-a12d059780c6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.012185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.028791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.029400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.030044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.030044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.030679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=3c56f73fdd7742d6a336f41ff9da3578) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:48.030981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.032583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.039113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 382/777] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:47 2022] GET /identity/v3/role_assignments?user.id=3c56f73fdd7742d6a336f41ff9da3578&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:48.166238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.178951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:48.183786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b5c9c2-bf26-4541-9056-765559c77d6b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.250583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.251624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.252106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.252588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.269486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:48.277172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:48.278324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:48.279166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:48.280079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 396/778] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:48.285237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.377210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2adfb-1036-412d-8969-7c71ca1a2d92 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.451698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.452311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.452545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.452816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] PATH_INFO: `/v3/services/network` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.453411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.454804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:48.469534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:48.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:48.471500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:48.479451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 383/779] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/services/network => generated 88 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:48.480836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f848d-970b-4527-8286-5cd252074a29 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.494224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.524326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.525774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.527081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.528220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.529091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:48.530851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.532233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.539448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:48.552404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 397/780] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:48.558158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.639977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8a9af7-2554-49c5-a52a-3d915fe0f80c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.686702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.687760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.688260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.688722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.689563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] RBAC: Adding query filter params (name=network) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:48.690086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.691674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.706196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 384/781] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/services?name=network => generated 124 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:48.706196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.720028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:48.854195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:48.864350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797c4fd9-dae2-4e1b-9dab-f903129c746e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.887139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a9ad4e-9724-4e7b-9376-dc793efc8125 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:48.933536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.934463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.934736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.935005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.951701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:48.952317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:48.952592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:48.952863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:48.953554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] RBAC: Adding query filter params (type=network) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:48.953857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.955236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.961504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 385/782] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/services?type=network => generated 350 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:48.963046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:48.964632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:48.968040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:48.992012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:48.992979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:48.994549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:48.994549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:48.994549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:48.994549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 398/783] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 435 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:49.002928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:49.090436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ca819f-faeb-4c80-9629-04ab90734f91 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.106460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:49.138365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.139415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.139891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.140371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.141266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] RBAC: Adding query filter params (interface=public, service_id=be212373e4b44721ba2a52a23207e5c2, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:49.141372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1147e712-6bad-44cf-ad23-d6e3975e084e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.141860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.143639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.156707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 386/784] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:48:48 2022] GET /identity/v3/endpoints?service_id=be212373e4b44721ba2a52a23207e5c2&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:49.162776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:49.176137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.176866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.177041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.177273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.177930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:49.178775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.186016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.191946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:49.192939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 399/785] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:49.198732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:49.352580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff289157-4332-4a9b-bc04-e2ac36b13765 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.385489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75dd0565-85c3-446d-849c-4b2274f91a65 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.410313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:49.418133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.418823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.419104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.419375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] PATH_INFO: `/v3/users/swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.433365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.435406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.443765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.444916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.445195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.445461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.446087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.447447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d97a6d5b-1b46-454b-b8d4-4ad84f478cad None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:49.456386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:49.457318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:49.458261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:49.460454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 387/786] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/services => generated 772 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:49.461085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 400/787] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/users/swift => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:49.466634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:49.622152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:49.648254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca3f4fc-5ff1-4f81-9a6f-d2587f01b3ec None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.689764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.690443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.690722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.690992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.691768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:49.692075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.694142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.720738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 388/788] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:49.721492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1827b-6e05-4eaa-a290-813fd5ee7d73 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.721785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1827b-6e05-4eaa-a290-813fd5ee7d73 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.722088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd1827b-6e05-4eaa-a290-813fd5ee7d73 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.723644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 401/789] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:49.726425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:49.731324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.731599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.731887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.751684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:49.753009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:49.830323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c29414-a37e-432c-8333-f213f049e2be None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:49.849331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 402/790] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:49.867426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:49.868061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:49.868339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:49.868611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:49.870568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:49.879480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:49.880889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:49.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:49.898850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:49.900176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:49.900176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:49.900176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:49.900176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 389/791] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:49.906119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.056220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae1840-67f2-48c9-846c-c42c4415f5bf None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.073607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:50.091380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.091995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.092274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.092550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.093194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:50.093490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.094927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2596ca8b-1d59-449c-bfb9-454f7d5bf415 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.107002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 403/792] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:50.112616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.151967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a3f2-97be-48a2-9eb7-d11339836780 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.152249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a3f2-97be-48a2-9eb7-d11339836780 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.152505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24a3f2-97be-48a2-9eb7-d11339836780 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.154139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 404/793] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:50.158953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.159236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.159515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.179805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:50.181402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:50.241725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea05f2-2668-4ef9-ab03-f55dcc13314d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.264476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 405/794] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:50.265357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.284757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.285402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.285682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.285955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.293829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.295337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:50.307096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:50.308001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:50.309254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 390/795] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:50.314039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.402481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:50.427074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79651f27-1de4-4b97-8aab-43a1e4fec8c1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.438327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:50.483547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c0d4ba-efdc-434a-a257-08e3ae1a284b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.503647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.504298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.504577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.504852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.523719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.524481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.524856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.525243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.526058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:50.526463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.528482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.534886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:50.540381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:50.541405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:50.542256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 406/796] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:50.547479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 391/797] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:50.548961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.554082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.684463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7f715-c348-4d66-af1a-5133dee656c0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.706974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a9813e-8d96-44d7-a53f-906deabf0232 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.715684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:50.736238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.736847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.737122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.737389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.738078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:50.738377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.739752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.743979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.744638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.744917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.745191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/0449fafc849145c8b752a0b028ad2679/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.747105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=0449fafc849145c8b752a0b028ad2679, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.748532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e538cc68-af2a-4b9a-9f10-fa3baea233fd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.753297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 407/798] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:50.754455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.763385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 392/799] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:50 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/0449fafc849145c8b752a0b028ad2679/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 10:48:50.764487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daaf785-c3dc-4293-8ab1-2d4f1c1940b5 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.764782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daaf785-c3dc-4293-8ab1-2d4f1c1940b5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.765059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daaf785-c3dc-4293-8ab1-2d4f1c1940b5 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.766582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 393/800] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:50.771127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daaf785-c3dc-4293-8ab1-2d4f1c1940b5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:50.793953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:50.879023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee86433-6151-416d-accb-f54156107f3e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.892691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daaf785-c3dc-4293-8ab1-2d4f1c1940b5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:50.920307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.921351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.921630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.921899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.922813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.924383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.934245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:50.935307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.935775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.936237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1cd7135f-dc46-40b5-a347-a8c8b18ccc7c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:50.936346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:50.937353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:50.938266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 408/801] 10.222.0.46 () {56 vars in 1129 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:50.939257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:50.939500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:50.939774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:50.960369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:50.962452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:50.970280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:50.971670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:50.987667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:50.988862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:50.990006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:50.990006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:50.990006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:50.992356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 394/802] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:50.993357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.020560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.070001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 409/803] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:51.070001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.096543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.168572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb32fb-c9ef-42eb-b9c6-efafa9d88099 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.183141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a5769-5b7c-4670-8de6-e05e444205a0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.218782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.219585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.219862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.220134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.220776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:51.221074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.222474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.236558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 410/804] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:51.242885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.269328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.269789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.270088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.270360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.270994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:51.271296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.272744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.283863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 395/805] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:50 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:51.289891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.378838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.399946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c3918c-18d9-47df-ae03-af587cc01899 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.415622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.436126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.436738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.437016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.437288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.437781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec2f871-74df-4e6c-9c02-c96f09a11743 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.437953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:51.438258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.439697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.445171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 396/806] 10.222.0.46 () {56 vars in 1124 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:51.445995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.477509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.478492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.478998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.479978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.489345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.491354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:51.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:51.507242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:48:51.508141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:51.511112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 411/807] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:51.512759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.559584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a8ae58-40ba-4016-9d7d-28b1b0f233b9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.594411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.595052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.595328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.595597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.607117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:51.612427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:51.613354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 397/808] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:51.614793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.624873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.652540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe87653-e351-4e03-bf9f-8f476cc9358c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.708172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.709075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.709433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.709710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.710413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.711828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.732225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:51.734124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be53d2e0-8146-425a-8cb0-127c5b94fb21 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.747053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 412/809] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:48:51 2022] POST /identity/v3/endpoints => generated 346 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:51.748024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.819538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.820180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.820461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.820719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.821490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:51.821797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.830181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.867495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 398/810] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:51.872729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.902379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a277ede0-4672-45ed-be9f-5c6603af8b98 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:51.939528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:51.940164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:51.940448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:51.940723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:51.941405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:51.941713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:51.943146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:51.953512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 413/811] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:51.959104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:51.983761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.048592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d5b2ee-f9dd-4ac4-987d-2e4c344d05c7 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.069464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.092065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.092963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.094195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.094658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.124523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cf21b-7acd-4fca-9d40-d9f7a1438685 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.175740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.176492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.176847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.177191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.178871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.181205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec5dbff-cd97-4a5c-8f9f-aca25214e7b4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.191393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.192939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-da76c3b1-bb12-4c46-9be6-e48552645846 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:52.211801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:52.212980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:52.214076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:52.214076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:52.214076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:52.215702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 414/812] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:52.218973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f01f560-6649-4a6f-86db-3ed388479b91 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.219399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f01f560-6649-4a6f-86db-3ed388479b91 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.219799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f01f560-6649-4a6f-86db-3ed388479b91 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.221271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 415/813] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:52.226856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f01f560-6649-4a6f-86db-3ed388479b91 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.276007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.326246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 399/814] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:51 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:48:52.327970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.328466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.328971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.348493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:52.350644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:52.353238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f01f560-6649-4a6f-86db-3ed388479b91 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.367585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.392356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.393022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.393303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.393574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.394263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:52.394566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.395937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.410019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 416/815] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:52.410589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.437832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 400/816] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:52.439000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.553960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e0044-3fe7-4c7e-9a85-753aee57c964 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.621847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.622508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.622783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.623048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.626363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.642163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.643606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.646658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28e2275-1a14-4914-8dcf-52408c5f7f5f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:52.671596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:52.672572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:52.674070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:52.674070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:52.674070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:52.674070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 417/817] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:52.679496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.722140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.722780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.723056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.723324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.725270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.737556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.739046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:52.758118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:52.760250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:52.762536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:52.770231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 401/818] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:52.770231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.810416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdaba74-38d2-4aa9-8e98-31f8077d9f82 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.844898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.845505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.845780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.846088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.846740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:52.847035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.848392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.861302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 418/819] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:52.862163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.876355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e3a724-cba0-4510-a48c-5de20f543cce None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:52.925729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:52.926664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:52.927330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:52.927626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] PATH_INFO: `/v3/services/network` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:52.929103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:52.931163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:52.943222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:52.945123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:52.946257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 402/820] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:52.947930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:52.956468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.970358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:52.974299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4afbca9-7f15-43ac-ae01-44b0be8b622b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.019755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.020374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.020699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.020978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.021750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.022086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.023742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.046352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 419/821] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.047072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.071450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feccafff-a72e-41c1-bd82-d15a993932e8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.114845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.116012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.116289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.116560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.126224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.127675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:53.143788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:53.144722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:53.145599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:53.149812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 403/822] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:53.151218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.170460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9d34f-fcdc-4f26-a27c-1bf81ef1af87 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.206301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.207214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.207677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.208137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.208967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] RBAC: Adding query filter params (name=network) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.209364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.210752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.220039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 420/823] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/services?name=network => generated 124 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.220979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.230274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.232088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.259683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08d906c-fce4-4d5a-8a92-3240a0b7cdb9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.301853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.302793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.303698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.303973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.315744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.317175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fe69e091-4a73-4fe0-ae8d-b7997f5e2701 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:53.331207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:53.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:53.332988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:53.332988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:53.332988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:53.333165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c24777-124c-4547-966d-e9645d31fe06 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.337003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 404/824] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:53.338259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b440db16-0609-473f-9f44-39f9d35be648 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.338536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b440db16-0609-473f-9f44-39f9d35be648 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.338803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b440db16-0609-473f-9f44-39f9d35be648 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.340231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 405/825] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:53.340761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b440db16-0609-473f-9f44-39f9d35be648 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.373895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.374542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.374820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.375092 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.375775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.376078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.377590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.390312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 421/826] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.391098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.450310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b440db16-0609-473f-9f44-39f9d35be648 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.463336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.476557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.486987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.488059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.488340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.488769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.489391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] RBAC: Adding query filter params (type=network) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.489688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.491098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c7b22-5999-4a60-b302-1add814db8b6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.500448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 406/827] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/services?type=network => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.500448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.500448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.500448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.504971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb56436a-447c-4873-9909-7f079a8c91ee None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.516405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:53.517575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:53.542406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.543077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.543354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.543619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.544250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.544547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.545886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.558731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 422/828] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.560167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.598213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 407/829] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:53.598213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.673786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281119f-069b-4ebd-811b-80dd4bbe8d5f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.702557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.711368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.712099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.712484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.712851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.713678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:53.714114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.715626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.730003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 423/830] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:53.730003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:53.766790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf4c8aa-40ac-4a5d-94f9-b05fd9b54248 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.925100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfeffc4-f9d6-446c-8912-b36fd989174a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:53.948859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:53.970138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.971475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.971475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.971475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.975189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:53.975818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:53.976098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:53.976374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:53.977053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.978474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.979684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:53.981375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:53.996162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:53.997156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 424/831] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:53.998082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.007128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 408/832] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:48:53 2022] POST /identity/v3/endpoints => generated 337 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:48:54.007885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.079117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.142859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c696f87a-5c03-4bd1-ab42-dbf8d3831e7d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.164784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc65dbca-853c-49d1-8f20-b2081ad92d1e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.187331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.188118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.188400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.188669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.189428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:54.189734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.191761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.210856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 409/833] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:54.213908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.214580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.214856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.215115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.216497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.226521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:54.232084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:54.233013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:54.234044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 425/834] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:54.239057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.258146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.353400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4129bbe-0633-49aa-bd3f-b1838452bfd0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.372476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289bbf9-a16d-4087-b7d7-3c06bdea7fcf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.383587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.390435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.391294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.391752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.392175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/3c56f73fdd7742d6a336f41ff9da3578/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.407098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=3c56f73fdd7742d6a336f41ff9da3578, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.408748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf98001c-247c-4d0d-8fa8-48360ebcfd4c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.408871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.409404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.409679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.409947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.410631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:54.410929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.412335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218512a0-8c10-4570-bbd5-ae44a0fceecf None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.420688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 426/835] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:54.421887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445fde8-abe6-4533-a174-7990e8758de7 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.422196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445fde8-abe6-4533-a174-7990e8758de7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.422469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445fde8-abe6-4533-a174-7990e8758de7 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.423799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 427/836] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:54.435877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4445fde8-abe6-4533-a174-7990e8758de7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.436117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 410/837] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:48:54 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/3c56f73fdd7742d6a336f41ff9da3578/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 10:48:54.436117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.436117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.436639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.463132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:54.465187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:54.553850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 411/838] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:54 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:54.575478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4445fde8-abe6-4533-a174-7990e8758de7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.612785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.613516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.613864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.614273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.626071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.627593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:54.642905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:54.644066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:54.645186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:54.645186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:54.645186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:54.647835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 428/839] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:54.651971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.666995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.761338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19fe712-23b8-4d1f-bfad-8808c9a5489a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.797432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.798107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.798340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.798612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.799264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:54.799567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.800941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.817219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 412/840] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:54.819826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.822934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:54.851754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.901139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:54.938160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9430fb-77e8-4941-9c45-ac50c717e659 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.955125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c19d4e4-7164-49e2-aeb5-e95ded49970f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:54.976672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:54.977589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:54.977826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:54.978239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:54.978933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:54.980659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:54.989942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:54.991209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:48:54.992318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:54.994237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 413/841] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:55.000270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.000909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.001192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.001468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.003062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.013419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.015604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:55.031496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:55.032456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 10:48:55.033387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:55.039891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 429/842] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:48:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:55.041948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.142299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:55.148659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7ee409-d89b-469a-9e9c-783e2f543b4f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.162140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e92c959-e4ed-4692-ab20-71406baa0368 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.191679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.192457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.192832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.193202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.194530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:55.195114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.196395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.198669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:55.206843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 414/843] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:55.208151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.210143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.210929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.211383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.212068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.213999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:55.214480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.216052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.246162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 430/844] 10.222.0.46 () {56 vars in 1135 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:55.247094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.321238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718b9189-0c43-402d-b27c-0f29485194e4 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.356247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.356965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.357168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.357433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.358095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:55.358400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.359755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.370005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 415/845] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:55.388811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0769e189-1ee4-43c3-9759-a21d93fa224d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.427180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668a4274-822c-401e-be99-3011c768dfff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.428326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a4274-822c-401e-be99-3011c768dfff None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.428695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a4274-822c-401e-be99-3011c768dfff None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.428972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a4274-822c-401e-be99-3011c768dfff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.442883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:55.505164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668a4274-822c-401e-be99-3011c768dfff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.507212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668a4274-822c-401e-be99-3011c768dfff None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.613680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 431/846] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:48:55.621046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.688352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:55.728183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b3da9-64a5-490b-84b8-bfe31bee7985 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.764155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.764802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.765084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.765355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.776096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.777565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:55.793180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:55.794180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:55.795071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:55.795071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:55.795071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:55.795071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 416/847] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:55.801458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668a4274-822c-401e-be99-3011c768dfff None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:55.846500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:55.905744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668a4274-822c-401e-be99-3011c768dfff None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:55.942455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:55.943449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:55.943952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:55.944417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:55.945304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:55.945809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:55.947470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:55.959852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 432/848] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:55.965509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:56.074629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc0db74-a984-4eb0-965b-ecfa04aab942 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:56.093685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:56.116148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:56.116814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.117147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.117430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.125232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:56.126722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-c7087567-2897-4ca2-a8e9-d692e665768e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:56.140791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:56.141699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:56.142587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 417/849] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:56.149282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:56.213114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b6e990-97fe-42b0-a213-0d358ea88cab None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.213310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b6e990-97fe-42b0-a213-0d358ea88cab None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.213577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b6e990-97fe-42b0-a213-0d358ea88cab None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.215238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 418/850] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:56.220550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.220835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.221115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.243024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:56.244510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:56.263519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59525187-ed29-4e6d-a871-73a5d793ca65 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:56.265582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:56.306141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:56.306750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.307026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.307292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.307979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:56.308286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:56.309716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:56.322984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 433/851] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:56.331178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:56.344238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 419/852] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:48:56.444466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3eb750-0f5e-44af-8cbe-97987af9eec0 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:56.480647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:56.481516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.482010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.482456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.483398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=0449fafc849145c8b752a0b028ad2679) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:56.483880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:56.485468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c6b205-4315-4234-ba63-8e36abc84c4c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:56.492015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 434/853] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:56 2022] GET /identity/v3/role_assignments?user.id=0449fafc849145c8b752a0b028ad2679&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:56.613424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:56.726106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5cd1b3-5b15-4844-8341-54c1fd76680a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:56.734187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62858c-2973-47b3-b076-70021711d9c9 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.734658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62858c-2973-47b3-b076-70021711d9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.734658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62858c-2973-47b3-b076-70021711d9c9 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.735130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:56.736247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 435/854] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:48:56.741958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.742286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.742557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.762944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:56.764145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:56.767272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:56.767947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.768228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.768502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.769090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:56.770491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:56.779594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:56.780547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:48:56.781437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:56.785524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 420/855] 10.222.0.46 () {56 vars in 1132 bytes} [Fri Oct 28 10:48:56 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:56.786740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:56.791762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:56.850039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 436/856] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:56.899599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c7cf98-2b68-428d-bd10-a076dad58855 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:56.935882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:56.936738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:56.937170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:56.937619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:56.938428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:56.938878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:56.940450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:56.950088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 421/857] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:48:56 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:56.952458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.060385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f878fa-e562-4b46-b726-15504e3f87a0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.098438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.099386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.100483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.100944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.101727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:57.102220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:57.103728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:57.112768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 437/858] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:48:56 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:57.119778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.161205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:57.238374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb0c12-4634-46fb-9465-5d9a739d47e3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.277330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.278004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.278295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.278571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.291282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:48:57.296738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:57.297709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:57.298562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 422/859] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:48:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:57.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.334259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:57.421504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0f9a68-d4b6-4e24-9ae2-baeb743012a1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.457639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.458681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.459709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.460187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.461088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:57.461612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:57.463279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88656d6-3166-4aeb-a6fe-25c60a790ba4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:57.473725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 438/860] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:48:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:57.482766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.585873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:57.592262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd18bb7-b843-4440-afb3-3737e6bca10e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.628243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.628886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.629168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.629438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.641191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:57.642774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:57.643960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399a423e-bf11-4796-b91f-ead96b54bd36 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.644253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399a423e-bf11-4796-b91f-ead96b54bd36 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.644525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399a423e-bf11-4796-b91f-ead96b54bd36 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.650149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 439/861] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:57.651748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.652034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.652309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:57.659480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:57.660428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:57.661275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:57.661275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:57.661275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:57.661275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 423/862] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:57.666620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.673706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:57.675497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:57.750353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 440/863] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:57 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:48:57.773150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb40ce7-4b94-415f-86f3-10c12350c678 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.808317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.808983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.809268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.809542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.810208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:57.810513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:57.811871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:57.824779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 424/864] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:57.830279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:57.846087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:57.936764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50cdd9-2948-4ee5-8789-7025f5752f98 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:57.972748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:57.973726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:57.974244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:57.974882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:57.982629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:57.984313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:48:57.996160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:57.997078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:48:57.998006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:48:58.000393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 441/865] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:48:57 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:58.006826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.013667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.016773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:58.093124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:58.127925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507dbf5-fdfe-40bc-b3e5-342343917f88 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:58.135336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d0d9d-ab55-4b7c-bb64-f0b17a48d096 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:58.165671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:58.166618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.167084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.167539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.168415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:58.169966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:58.172519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:58.173291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.173766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.174222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.175224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:58.175731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:58.177341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:58.190021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 442/866] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:48:58 2022] POST /identity/v3/services => generated 251 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:48:58.203761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 425/867] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:48:58 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:58.210289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.346603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711b02c6-f254-4f60-9e94-0a9f524bae5f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:58.376562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:58.386820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:58.387687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.388155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.388609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.461376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:58.463147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:58.565277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 443/868] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:48:58 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:48:58.572303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.641406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:58.684866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918590dc-4a41-4bab-9cb8-a76cec87f23e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:58.720021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:58.720658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.720933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.721201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.736424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:58.737881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7da8c2bb-9d10-422c-be39-fe6135bde514 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:48:58.752957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:58.753931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:48:58.754848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:48:58.754848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:48:58.754848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:58.754848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 426/869] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:48:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:58.761777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.773498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062f64c-6b81-4efc-a6da-5b6699f00174 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.773801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062f64c-6b81-4efc-a6da-5b6699f00174 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.774080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3062f64c-6b81-4efc-a6da-5b6699f00174 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.775646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 427/870] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:58.781336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.781621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.781896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.795998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:58.807385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:58.808580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:58.874212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ec39f-a856-40b6-8649-d045e9e8b563 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:58.890054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 428/871] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:58 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:48:58.918090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:58.919032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:58.919536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:58.920028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:58.920946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:58.921451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:58.923148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:58.936219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 444/872] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:58.942087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:58.993524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:59.051655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065b59d-55e8-4a0c-90f4-6db6231f18f5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.089579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.090245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.090524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.090794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.097720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.099195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:48:59.110899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:59.111812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:59.112679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 429/873] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:48:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:48:59.119470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.199643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.220425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:59.234769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337aceed-c13f-40b1-b1c2-354d0d346059 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.272847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.273832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.274497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.274771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.275460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:59.275909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.277632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.292404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 445/874] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:48:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:59.298404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.334945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269eb09-9df3-4c62-a78d-fc7b70972224 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.343671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:59.374524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.375442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.375872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.376306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.377034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.378675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:48:59.389125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:48:59.390357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:48:59.391300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:48:59.398401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 430/875] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:48:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:48:59.400654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.417516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1188c4a8-ede0-4eef-8cee-a537df1c7497 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.456437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.456970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.457251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.457523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.458293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=3c56f73fdd7742d6a336f41ff9da3578) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:59.458603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.460883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d9dd68-00be-4434-811b-28b20c2af4a4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.474018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 446/876] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:48:59 2022] GET /identity/v3/role_assignments?user.id=3c56f73fdd7742d6a336f41ff9da3578&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:59.474018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ae3dc-3b9a-4eff-ab70-2f684640facb None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.474018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ae3dc-3b9a-4eff-ab70-2f684640facb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.474018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7ae3dc-3b9a-4eff-ab70-2f684640facb None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.474018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 447/877] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:48:59.476274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.476562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.476840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.497202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:48:59.498526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:48:59.508433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:59.530711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f426a94-e55d-4637-8732-507356277c73 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.579157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.579926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 448/878] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:48:59 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:48:59.580334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.580677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.580892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.581550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:59.581854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.583292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.592835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 431/879] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:48:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:48:59.602066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.709992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af246a4c-d5dc-404f-b644-94fe6b943b55 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:48:59.745544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:48:59.746190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:48:59.746471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:48:59.746745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:48:59.747382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:48:59.747686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:48:59.749045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3377ee50-c1e3-4d78-a78d-3bd98850bfe0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:48:59.753958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 449/880] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:48:59 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:48:59.855277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:48:59.893069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:48:59.971928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddd8c8c-b237-4ce6-927d-32bb4ce9e2ef None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:00.008820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:00.009434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:00.009709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:00.009989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:00.010640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:00.012000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8836b2-b9d2-44a2-b9f1-4349105f8af6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:00.032244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 432/881] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:48:59 2022] POST /identity/v3/services => generated 269 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:00.548083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e8c42-dada-414d-b736-b45bb7fd2fc8 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:00.548592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e8c42-dada-414d-b736-b45bb7fd2fc8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:00.548592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e8c42-dada-414d-b736-b45bb7fd2fc8 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:00.550130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 450/882] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:00.556013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:00.556410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:00.556592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:00.575198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:00.576377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:00.650413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 433/883] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:00 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:00.923468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628e8c42-dada-414d-b736-b45bb7fd2fc8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:00.933451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:01.035502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628e8c42-dada-414d-b736-b45bb7fd2fc8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:01.070684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:01.071318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.071594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.071864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.072444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:01.073798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:01.082641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:01.085763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:01.091003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 451/884] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:00 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:01.092951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:01.115739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:01.195586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d26fa07-db78-405b-8004-825a91f1b8dc None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:01.230607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:01.231218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.231497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.231767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.232397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:01.232698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:01.234089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:01.239082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 434/885] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:01 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:01.244993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:01.322686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:01.357132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8506bf7-05b8-485a-8bb8-f76d2172dae1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:01.392735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:01.393715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.394238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.394743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.395592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:01.396155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:01.397848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:01.403493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 452/886] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:01 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:01.409632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:01.495645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:01.512944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a72b8-2f58-4656-a315-e2178439bb0d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:01.547748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:01.548722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.549259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.549826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.550810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] RBAC: Adding query filter params (interface=public, service_id=75c193bf8f6c4077b12d7ced67e97f2d, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:01.551326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:01.553430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5880f1-91c5-40f2-9f16-7453c604f8f2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:01.559118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 435/887] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:49:01 2022] GET /identity/v3/endpoints?service_id=75c193bf8f6c4077b12d7ced67e97f2d&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:01.565332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:01.669272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f166b67a-4053-4f03-829f-9f2cf08592b3 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:01.704334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:01.704945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.705235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.705546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.706183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:01.707595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287cbe1d-7dd5-4e7f-a0e3-84167cc17ca4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:01.708591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:01.714111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 453/888] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:01 2022] GET /identity/v3/services => generated 1268 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 10:49:01.868282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd517bd4-5c43-44f2-8f25-37975aa1d0e3 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.868686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd517bd4-5c43-44f2-8f25-37975aa1d0e3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.868895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd517bd4-5c43-44f2-8f25-37975aa1d0e3 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.870445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 436/889] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:01.876284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19393c6b-1807-409c-bc49-25b9835ae162 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:01.876584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19393c6b-1807-409c-bc49-25b9835ae162 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:01.876846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19393c6b-1807-409c-bc49-25b9835ae162 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:01.896311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-19393c6b-1807-409c-bc49-25b9835ae162 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:01.898119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-19393c6b-1807-409c-bc49-25b9835ae162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:01.978247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 454/890] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:01 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:02.037831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f7648-350f-4844-a73b-b0d091bc029f None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.038186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f7648-350f-4844-a73b-b0d091bc029f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.038405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0f7648-350f-4844-a73b-b0d091bc029f None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.039987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 437/891] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:02.045273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5b630-b690-4a23-bd1f-69916da3755c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.045568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5b630-b690-4a23-bd1f-69916da3755c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.045846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5b630-b690-4a23-bd1f-69916da3755c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.064784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2ca5b630-b690-4a23-bd1f-69916da3755c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:02.066075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2ca5b630-b690-4a23-bd1f-69916da3755c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:02.142178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 455/892] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:02 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:02.256724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f7648-350f-4844-a73b-b0d091bc029f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.342607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71645c-9b5a-4401-b4ae-6f323ee8dc11 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.342958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71645c-9b5a-4401-b4ae-6f323ee8dc11 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.343153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc71645c-9b5a-4401-b4ae-6f323ee8dc11 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.344480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 456/893] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:02.349450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.350304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.350578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.367590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0f7648-350f-4844-a73b-b0d091bc029f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:02.369634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:02.370867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:02.411864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:02.413014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.413299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.413851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.426862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:02.432293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:02.433402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:02.434475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 438/894] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:02.435245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:02.438909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.450361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 457/895] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:02 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:02.450641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:02.451449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.559802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edea0356-2f17-4e3c-be51-32479871c9ee None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:02.582153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cfc57-fcb5-4387-848f-0dd460c13059 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:02.598770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:02.600999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.601746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.602255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.603393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:02.604269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:02.606128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:02.618806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 458/896] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:02.626976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.628870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:02.629730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.630127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.630473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.631277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:02.632650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:02.650353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 439/897] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:49:02 2022] POST /identity/v3/services => generated 241 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:02.694144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:02.735929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.757692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71963e8d-480f-4e5d-965a-53ab17a51d4c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:02.808738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:02.809805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.810429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.811023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.817166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:02.826896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:02.828666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:02.845594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:02.846626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:02.847519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:02.847519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:02.847519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:02.847519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 459/898] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:02.853003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.865826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dce559-02bd-4af5-8b1c-952cc27d380c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:02.908738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:02.909840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:02.910408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:02.910684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:02.911731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:02.914271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:02.927550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:02.928599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:02.929483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 440/899] 10.222.0.46 () {56 vars in 1132 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:02.935156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:02.973412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:02.979260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce88e66-b627-4899-89d0-5acaeccb3f0b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.022614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.023234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.023512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.023783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.024435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:03.024737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.026172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.044298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 460/900] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:03.045562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.067003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c52738-befa-4cef-8648-e2153a0b4f61 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.105813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.111235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.112337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.112569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.113008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.113820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:03.114317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.116131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.121746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 441/901] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:02 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:03.129470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.166803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c69ca-93ac-4851-a279-da3ba2d3f821 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.209872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.210558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.210836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.211156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.219001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.220491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:03.235126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:03.236086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:03.237047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:03.240999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 461/902] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:03.242131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.246010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1ba3be-aab7-4acd-a9a7-ca8928f23e69 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.265207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.287556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.288167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.288447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.288718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.289589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:03.289700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.291026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.300951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 442/903] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:03.302300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.371441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.379291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fcd6e4-0713-45fe-adc5-1fe007d070e9 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.417870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.418673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.418960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.419285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.420136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:03.420446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.421881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.427500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d19c3-22e3-476f-8948-8df1fafafa6c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.449818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 462/904] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:03.451396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.478093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.479476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.480492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.480993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.482575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] RBAC: Adding query filter params (interface=public, service_id=0ba46aefabfd491ebed6e92842e97e78, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:03.483361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.485637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.492990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 443/905] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/endpoints?service_id=0ba46aefabfd491ebed6e92842e97e78&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:03.498906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.545186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.570439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f9b65d-aaae-44c0-8590-9e07f7202c87 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.611889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.612556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.612858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.613143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.627775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ac6666-07ce-4fc5-b095-2b59547771d9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.660652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.677812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c450025-1553-4012-900a-8427c556384a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.678587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c450025-1553-4012-900a-8427c556384a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.678961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c450025-1553-4012-900a-8427c556384a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.679307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c450025-1553-4012-900a-8427c556384a None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.679984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c450025-1553-4012-900a-8427c556384a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.681782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c450025-1553-4012-900a-8427c556384a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.688197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.689858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d550a2-974e-4fcb-8709-8ee2aa437890 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.694717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 444/906] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/services => generated 1497 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 10:49:03.775279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:03.801153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 463/907] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:03.806930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c450025-1553-4012-900a-8427c556384a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:03.923915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c450025-1553-4012-900a-8427c556384a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:03.960020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:03.960659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.960934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.961199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.971820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:03.973784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0045c-43d9-420b-bc4a-f0a774e66dd7 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.974091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0045c-43d9-420b-bc4a-f0a774e66dd7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.974361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0045c-43d9-420b-bc4a-f0a774e66dd7 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.975165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:03.976201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 464/908] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:03.981524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:03.981815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:03.982102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:03.991089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:03.992053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:03.993339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:03.993339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:03.993339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:03.993339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 445/909] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:03.999548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.000815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:04.002021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:04.051822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:04.080814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 465/910] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:04.120137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad21bf5-af6b-4d86-9e75-3ed7e70c6740 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.155186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.156078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.156569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.157035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.157856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:04.158369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.159925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.176863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 446/911] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:04.178989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.285310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04a306-f9c4-490a-86ac-b388e390c3a4 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.293087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:04.323067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.323720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.324002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.324280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.331138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.332591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:04.344080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:04.345071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:04.345936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:04.350036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 466/912] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:04.351385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.369101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.467945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:04.472001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e3bb2-ac11-4d1c-aa7c-95a7e1e44f02 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.496725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878fdc8-516a-4d14-9b6a-22c46ed7a739 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.512870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.513840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.514361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.514826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.515728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:04.516237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.517858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.530484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 447/913] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:04.537469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.545527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.546474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.546969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.547464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.548336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.550033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:04.560749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:04.561695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 10:49:04.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:04.569966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 467/914] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:04.571073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.605609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:04.656974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d31546-4edc-4076-8acf-c11e409ff2b5 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.695794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.696789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.697267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.697754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.698712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] RBAC: Adding query filter params (role.id=47f258b925fe4809989a40c4b5c9472a, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=3c56f73fdd7742d6a336f41ff9da3578) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:04.699216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.700957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.705525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c94607-70ad-44a7-bb53-0de72b59d79a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.707624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 448/915] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/role_assignments?user.id=3c56f73fdd7742d6a336f41ff9da3578&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=47f258b925fe4809989a40c4b5c9472a => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:04.748654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.749711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.750321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.750821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.751835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:04.752373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.754167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cde923-1bb9-462d-933a-0d5c617fc7b9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.760714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 468/916] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:04.767051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.797354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:04.875653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd984f6-065e-4638-a909-00f9da82cb5f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.875854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd984f6-065e-4638-a909-00f9da82cb5f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.875975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd984f6-065e-4638-a909-00f9da82cb5f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.877612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 469/917] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:04.883995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.884263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.884610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.888999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9aecde-ff7d-48e7-aa4b-6ca22583d913 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:04.904242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:04.905446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:04.932817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:04.933619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:04.934004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:04.934450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:04.935170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:04.935801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:04.937092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:04.942552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 449/918] 10.222.0.46 () {56 vars in 1114 bytes} [Fri Oct 28 10:49:04 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:04.948698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:04.996413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 470/919] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:05.043144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:05.057035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3addb5a5-6062-480d-aa0e-b095f6757954 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:05.092776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:05.093385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.093657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.093923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.094628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:05.095987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:05.120072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 450/920] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:04 2022] POST /identity/v3/endpoints => generated 346 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:05.264423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:05.292911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:05.374470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4b1d9-1603-4701-bcfe-cef87a9ca9ea None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:05.415049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:05.415740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.416023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.416300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.416879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:05.418272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:05.427405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:05.429039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:05.429843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 471/921] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:05 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:05.435355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:05.547244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5aa578-6326-48c7-8354-e75a6616a4e4 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:05.583249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:05.584037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.584241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.584508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.585151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:05.585509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:05.586923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:05.592645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 451/922] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:05 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:05.599084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:05.688945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:05.706829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e84b3a-feb5-45b6-a353-329dec65cf8e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:05.741249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:05.741913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.742218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.742490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.743162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:05.743465 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:05.744823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:05.750035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 472/923] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:05 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:05.756384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:05.768458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:05.868845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bcb3c8-e59b-4852-94e8-7aa50be2fde3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:05.904000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:05.904601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.904876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.905140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.905837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] RBAC: Adding query filter params (interface=public, service_id=0becf4f28f8342c1ae94b225e37532af, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:05.906168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:05.907524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f13b1-11b2-4716-96da-709b84a1c7f5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:05.913576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 452/924] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:49:05 2022] GET /identity/v3/endpoints?service_id=0becf4f28f8342c1ae94b225e37532af&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:05.919138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:05.964256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b138181-cee7-435c-a9ed-fdaa833cebf2 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.964577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b138181-cee7-435c-a9ed-fdaa833cebf2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.964799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b138181-cee7-435c-a9ed-fdaa833cebf2 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.966421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 453/925] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:05.974714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:05.975118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:05.975534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:05.994651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:05.996003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:06.039819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3766cde4-cc0c-47ab-bb95-848b9a60d2fa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:06.076011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:06.076974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:06.077456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:06.077956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:06.078810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:06.080400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:06.082054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 454/926] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:06.090696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 473/927] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:05 2022] GET /identity/v3/services => generated 1497 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 10:49:06.358141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:06.360320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:06.461841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e480922-f97f-4439-8e04-6ad90a7393e9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:06.497681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:06.498347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:06.498625 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:06.498900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:06.499482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:06.500850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:06.509767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:06.510723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:06.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 455/928] 10.222.0.46 () {56 vars in 1132 bytes} [Fri Oct 28 10:49:06 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:06.519346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:06.535814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:06.627329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844ac73-c434-4620-bf2b-4efb5a78bc68 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:06.662609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:06.663612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:06.664085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:06.664568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:06.665384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:06.665879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:06.667466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:06.672868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 474/929] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:06 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:06.683020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:06.749698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:06.785126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204fc5cb-f593-4ecb-8d91-fdac7bc94f91 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:06.820248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:06.820794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:06.821118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:06.821391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:06.822048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:06.822352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:06.823717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb3e0b7-1887-4db9-afe9-40cc19bdd785 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:06.828861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 456/930] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:06 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:06.834511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:06.922484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:06.946695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eeb44a-4325-487c-b40a-c8dcdeddbf0f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:06.984240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:06.984862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:06.985148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:06.985424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:06.986142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:06.987503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58898c-a459-43f3-a730-7a9a50aabf39 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:07.010815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 475/931] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:06 2022] POST /identity/v3/endpoints => generated 347 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:07.023132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356a667-1685-4e39-b5ea-3a0b67ed1ca9 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.023453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356a667-1685-4e39-b5ea-3a0b67ed1ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.023688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b356a667-1685-4e39-b5ea-3a0b67ed1ca9 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.025074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 457/932] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:07.031194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd98340-3597-4028-b52c-73f92dfbfc9d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.031502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd98340-3597-4028-b52c-73f92dfbfc9d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.031763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd98340-3597-4028-b52c-73f92dfbfc9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.051785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-bfd98340-3597-4028-b52c-73f92dfbfc9d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:07.053066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bfd98340-3597-4028-b52c-73f92dfbfc9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:07.137492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 476/933] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:07.411107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b356a667-1685-4e39-b5ea-3a0b67ed1ca9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:07.521331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b356a667-1685-4e39-b5ea-3a0b67ed1ca9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:07.551183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:07.557877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:07.558546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.558828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.559103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.567726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6849-ad00-4b55-a908-8301c2be57fc None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.568321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6849-ad00-4b55-a908-8301c2be57fc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.568801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea6849-ad00-4b55-a908-8301c2be57fc None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.570254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:07.571954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 477/934] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:07.575237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:07.576165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:07.577351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 458/935] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:07.579692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.580179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.580648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.582208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:07.600470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:07.602039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:07.680736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:07.683094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 478/936] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:07.703775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6594f1c2-09dc-494b-b239-59961174082b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:07.740845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:07.741808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.742369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.742849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.743831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:07.744338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:07.745894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:07.754686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 459/937] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:07.760636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:07.866989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2478a2b2-2c48-46cd-8906-85829a22dc8d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:07.903810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:07.904818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:07.905333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:07.905810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:07.917208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:07.919023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:07.934600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:07.935571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:07.936441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:07.936441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:07.936441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:07.938585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 479/938] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:07.944775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:07.976469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.009496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.026006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.075344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ba09a-fa83-43d2-b4f9-c17bf02e9e81 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.100127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7d8e53-d8ea-409d-b1b8-31354570c9cc None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.123537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.124273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.124646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.124991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.125718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:08.126129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.128752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.144983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.145955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.146561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.147006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.147893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.149508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.150876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 460/939] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:08.151874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.153335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING keystone.server.flask.common [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 28 10:49:08.173553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.sql.core [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 10:49:08.173553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 10:49:08.173553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [parameters: {'id': '58022b3e81f34732b989f0fad46acef7', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 10:49:08.173553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81305) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 58022b3e81f34732b989f0fad46acef7.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 58022b3e81f34732b989f0fad46acef7. Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:49:08.183451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result.read() Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 28 10:49:08.184278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application yield resource Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise self.value Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application session.commit() Oct 28 10:49:08.185264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.session.flush() Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application compat.raise_( Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 10:49:08.186177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 10:49:08.187349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application result.read() Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:49:08.188572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [parameters: {'id': '58022b3e81f34732b989f0fad46acef7', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.191033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:08.192744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 58022b3e81f34732b989f0fad46acef7. Oct 28 10:49:08.195830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.202171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 480/940] 10.222.0.46 () {60 vars in 1140 bytes} [Fri Oct 28 10:49:07 2022] POST /identity/v3/projects => generated 262 bytes in 222 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:08.205563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.270996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1e565-e719-4e3b-8c1a-39fa53a6855d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.287512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.310577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.311203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.311487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.311760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.320123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.321599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.322586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.330382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f506581-27bd-4ed2-936c-e7e457e6f25f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-4c63e8a2-7d5f-4fac-8209-1a494d463fb5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:08.334060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:08.335360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:08.336631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 461/941] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:08.340620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a309b2-7acb-4113-ae2c-30c40efb4557 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.340907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a309b2-7acb-4113-ae2c-30c40efb4557 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.341186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a309b2-7acb-4113-ae2c-30c40efb4557 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.342621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 462/942] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:08.343172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a309b2-7acb-4113-ae2c-30c40efb4557 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.376562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.377196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.377469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.377736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.384702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.386362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-858b3b3d-ddb4-4b99-be9c-480bb30cf15a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:08.397530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:08.398582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 481/943] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.404401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.405118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.421686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:08.423245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:08.456683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a309b2-7acb-4113-ae2c-30c40efb4557 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.491971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.492601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.492882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.493156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.493877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:08.494214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.495637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.502155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 482/944] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:08.502155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.523337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 463/945] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:08.524123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.555195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.606615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.627354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fb3a3a-a377-4162-a255-6c1c17aac1ff None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.633683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab91fa2-b6c6-4413-a5ba-837d0b97fe03 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.670875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.671997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.672426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.672869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.673679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:08.674160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.675777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.675894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e5290-4023-43f6-b101-f301439853af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.676613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e5290-4023-43f6-b101-f301439853af None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.676995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e5290-4023-43f6-b101-f301439853af None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.677366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e5290-4023-43f6-b101-f301439853af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.689009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801e5290-4023-43f6-b101-f301439853af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.690627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801e5290-4023-43f6-b101-f301439853af None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.706794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 483/946] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-801e5290-4023-43f6-b101-f301439853af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:08.716063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:08.717315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:08.718456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:08.718456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:08.718456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:08.719837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 464/947] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:08.726308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.793193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e5290-4023-43f6-b101-f301439853af None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.852075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6f1a9-f838-41b9-bd4d-fd6999edbb64 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.897582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.898353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.898710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.899061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.899784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:08.900160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.903089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.903701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.920306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 484/948] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:08.926934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:08.929861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e5290-4023-43f6-b101-f301439853af None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:08.937057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:08.977550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:08.978889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:08.979395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:08.981347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:08.982058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:08.983500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:08.996227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:08.997254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:08.998564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 465/949] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:09.003605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.050700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144311c-db49-4223-b2c7-3efc0adb43c7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.092870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.093874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.094383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.094832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.103607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.106171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:09.121492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:09.122505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:09.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:09.126093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 485/950] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:09.132601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.139308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33417c-9ca3-48ec-bee6-a379b082b68e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.177168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.178131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.178559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.179012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.179797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:09.180254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.181771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.187426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 466/951] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:09 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:09.193824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.211959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:09.215232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:09.257652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c09f096-d94b-4607-bec6-77c39fc87ba7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.298287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.299006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.299282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.299664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.300353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:09.300655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.302100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9720f3c-5db0-442d-b067-89394b55e8a8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.314677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9abd5-be71-460b-9001-4097b7aa857e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.317279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 486/952] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:09.318708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a012b-68ad-4036-9a67-759ee7a5dcaf None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.319139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a012b-68ad-4036-9a67-759ee7a5dcaf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.319556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19a012b-68ad-4036-9a67-759ee7a5dcaf None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.321157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 487/953] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:09.322880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a012b-68ad-4036-9a67-759ee7a5dcaf None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.354027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.354643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.355413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.355413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.356046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:09.356366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.357699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7284f54c-3198-4cc0-a0d0-a039705d3181 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.363449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 467/954] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:09 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:09.364588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.364876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.365791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.383965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:09.385191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:09.434929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19a012b-68ad-4036-9a67-759ee7a5dcaf None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.462148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 468/955] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:09.462148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.463896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:09.468870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:09.482554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.483218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.483492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.483759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/3c56f73fdd7742d6a336f41ff9da3578/roles/47f258b925fe4809989a40c4b5c9472a` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.487959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=3c56f73fdd7742d6a336f41ff9da3578, role_id=47f258b925fe4809989a40c4b5c9472a)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.489383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.513636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 488/956] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:49:09 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/3c56f73fdd7742d6a336f41ff9da3578/roles/47f258b925fe4809989a40c4b5c9472a => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:49:09.601194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f894234f-3559-4b33-9bf5-fcf60c7699ed None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.642540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.643432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.643885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.644313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.645142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.646805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.670972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 469/957] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:09 2022] POST /identity/v3/endpoints => generated 349 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:09.760914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:09.876342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d60e2b1-0a10-4894-979a-4872381c7e0d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:09.920477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:09.921799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:09.922292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:09.922735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:09.923523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:09.925089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:09.935219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:09.936623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:09.937866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:09.945508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 489/958] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:09 2022] GET /identity/v3/services/compute => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:09.947041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:10.011683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:10.059328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c552b7d-4911-465b-bb7b-1dcd895d6734 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:10.095890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:10.096500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:10.096772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:10.097041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:10.097668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:10.097964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:10.099370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98432040-5e70-46fb-9327-2e36fbaa5a3c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:10.104550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 470/959] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:09 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:10.110520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:10.164034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:10.223011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d6b41-3e41-4906-918e-24caa7f9f2a6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:10.259378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:10.260365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:10.260904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:10.261374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:10.262539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:10.263335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:10.266444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c38d76f-dca9-4f03-943b-a9586d333133 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:10.278677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 490/960] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:10 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:11.015662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c5aeb5-301b-4304-8811-2f8aa1e7b725 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.016181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c5aeb5-301b-4304-8811-2f8aa1e7b725 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.016181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c5aeb5-301b-4304-8811-2f8aa1e7b725 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.017645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 471/961] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:11.023541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.023939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.024108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.043323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:11.044501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:11.113514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 491/962] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:11.386564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c5aeb5-301b-4304-8811-2f8aa1e7b725 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:11.490959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c5aeb5-301b-4304-8811-2f8aa1e7b725 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:11.516570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:11.529029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:11.529568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.529842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.530134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.530788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:11.532210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:11.534744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING keystone.server.flask.common [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 28 10:49:11.548470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.sql.core [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 10:49:11.548470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 10:49:11.548470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [parameters: {'user_id': 'd0a6d05d55514f68be918e7e2b698422', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 10:49:11.548470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81306) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-bdd7761f-112f-4a2d-8c41-e6dfe27aada9 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application result.read() Oct 28 10:49:11.566187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:49:11.567104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application yield resource Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application next(self.gen) Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise self.value Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application session.commit() Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 10:49:11.568095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self.session.flush() Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application compat.raise_( Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 10:49:11.569050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 10:49:11.569941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application conn.query(q) Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application result.read() Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 10:49:11.570823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [parameters: {'user_id': 'd0a6d05d55514f68be918e7e2b698422', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:11.571659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 28 10:49:11.572432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:11.573348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 472/963] 10.222.0.46 () {60 vars in 1131 bytes} [Fri Oct 28 10:49:11 2022] POST /identity/v3/users => generated 160 bytes in 182 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:11.582011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:11.629537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:11.696155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a7879-9e81-4d66-974e-e6549f2ca140 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:11.733351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:11.734008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.734297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.734569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.742015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:11.743461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:11.755036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:11.756042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:11.756972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 492/964] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:11.757591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa3f19e-694a-482f-a6ea-266e7be971e8 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.757591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa3f19e-694a-482f-a6ea-266e7be971e8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.757591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa3f19e-694a-482f-a6ea-266e7be971e8 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.763776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:11.765236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 473/965] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:11.766399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1feddaf-0389-4714-91ad-3666d3bd7ac3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.766681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1feddaf-0389-4714-91ad-3666d3bd7ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.766950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1feddaf-0389-4714-91ad-3666d3bd7ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.785814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c1feddaf-0389-4714-91ad-3666d3bd7ac3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:11.787855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c1feddaf-0389-4714-91ad-3666d3bd7ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:11.873272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 474/966] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:11.882028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14c4b6-f653-4ea7-bf84-5b1ac279d664 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:11.883575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:11.920558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:11.921176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.921454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.921725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.922458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:11.922759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:11.924204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:11.938386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dc42e-c29b-48db-939b-599b6f824c5f None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.938991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dc42e-c29b-48db-939b-599b6f824c5f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.939475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321dc42e-c29b-48db-939b-599b6f824c5f None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.941383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 475/967] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:11.946637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577f229-9948-401b-86fd-44995d938390 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:11.946920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577f229-9948-401b-86fd-44995d938390 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:11.947192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1577f229-9948-401b-86fd-44995d938390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:11.960482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 493/968] 10.222.0.46 () {56 vars in 1101 bytes} [Fri Oct 28 10:49:11 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:11.967877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1577f229-9948-401b-86fd-44995d938390 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:11.969115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1577f229-9948-401b-86fd-44995d938390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:12.046822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 476/969] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:12.148193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.254605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f597911d-c209-48a7-9647-40d3228ad369 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.293925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.294583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.294857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.295124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.305798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:12.310539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:12.311475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:12.312309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 494/970] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:12.317817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1577f229-9948-401b-86fd-44995d938390 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.318591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.337239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.366676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.424998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1577f229-9948-401b-86fd-44995d938390 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.430186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6dba3d-ee33-419f-8a2a-c1db1288d0bc None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.461739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.462413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.462688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.462958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.463640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:12.463939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.465487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.470012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.470679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.470960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.471248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.471845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.473229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.478188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 477/971] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:12.481284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:12.487446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:12.488423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:12.489229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 495/972] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:12.494624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.587256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e327e84-2871-4256-8cb9-75413b3a7599 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.590047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.596184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.609798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396c19f0-6438-44c7-8124-c3d83cad74c7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.621406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.622061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.622341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.622611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.632984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.634540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.642446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.643075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.643355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.643627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.644254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:12.644550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.645881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb099d-e010-433a-a8a2-557e522345e3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-f66dcc73-aacd-45b6-aa21-e7fad8e2447b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:12.648535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:12.649436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:12.650342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:12.650342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:12.650342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:12.650342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 478/973] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:12.650941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6231ef-0ba0-46cf-bce3-95243229cd99 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.651223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6231ef-0ba0-46cf-bce3-95243229cd99 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.651491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6231ef-0ba0-46cf-bce3-95243229cd99 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.652809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 479/974] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:12.655935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 496/975] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:12.656940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6231ef-0ba0-46cf-bce3-95243229cd99 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.658247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.658534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.658803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.676718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:12.677924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:12.742673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 497/976] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:12.743483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.764660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6231ef-0ba0-46cf-bce3-95243229cd99 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.800323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.800930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.801233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.801503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.802175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:12.802529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.803876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.815444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 480/977] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:12.821358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.827565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.843067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:12.855834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d2a84c-94be-45fa-ae8a-de55ba7c2416 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.893930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.894588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.894860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.895134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.895757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:12.896058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.897998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.908672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 498/978] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:12.909630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:12.930020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d081f67-2f29-4853-97a5-88c950a6993c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:12.968364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:12.968988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:12.969273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:12.969541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:12.976897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:12.978354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:12.991190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:12.992125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 481/979] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:12.996826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.022232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e19a3-c1fe-4305-b016-c4610c83f2e8 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.054639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.055237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.055512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.055781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.056481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] RBAC: Adding query filter params (interface=admin, service_id=0becf4f28f8342c1ae94b225e37532af, region_id=RegionOne) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:13.056786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.058165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.065158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 499/980] 10.222.0.46 () {56 vars in 1249 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/endpoints?service_id=0becf4f28f8342c1ae94b225e37532af&interface=admin®ion_id=RegionOne => generated 193 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:13.066008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.069908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:13.078237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:13.097791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c440e20-34e2-4fac-bcfe-19227161468a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.130416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.131614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.131890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.132156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.132868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:13.133169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.134621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.151989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 482/981] 10.222.0.46 () {56 vars in 1149 bytes} [Fri Oct 28 10:49:12 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:13.152773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.168458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506317c6-1cc4-4577-aad1-1df927498c0d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.204540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.205148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.205429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.205703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.206376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.207926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.221909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 500/982] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:49:13 2022] POST /identity/v3/services => generated 242 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:13.222726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.260270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a69f74-027c-4b42-81e4-cd8f33634724 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.305020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:13.310794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.311393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.311731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.312003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.312721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.314094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.315465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:13.327879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 483/983] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:13 2022] GET /identity/v3/services => generated 1727 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 10:49:13.351950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b53d047-c572-4e53-a12b-ee3a68492a58 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.390794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.391435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.391714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.391985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.469527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.471444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.539498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:13.581211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 501/984] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:13 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:13.587492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.689419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a198cf-234f-4c17-a18f-9391581b4710 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.724417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.725052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.725328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.725599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.737303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.738774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:13.753347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:13.754523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:13.755443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:13.755443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:13.755443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:13.755443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 484/985] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:13.762110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.862910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f92f528-ebc4-4d0d-9751-29e31d9b3921 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:13.897085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:13.897702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:13.898006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:13.898282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:13.898928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:13.899233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:13.900592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:13.916389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 502/986] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:13.917651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:13.970924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:14.023566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a232304-18b3-4710-b4c6-289e9908452a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:14.057116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:14.057750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:14.058061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:14.058334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:14.065013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:14.066504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:14.078223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:14.079518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:14.080398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 485/987] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:14.085120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:14.110944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:14.188750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b05201-8c9d-43e3-a3bd-d52bde98b2c6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:14.223903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:14.224466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:14.224744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:14.225014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:14.225700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:14.226022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:14.227440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09bc9c9-4236-484a-81c8-f24e024c05ba None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:14.240394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 503/988] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:14.246984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:14.351855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1974b3f0-16f9-4e0b-bcf7-72897a8483ec None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:14.362524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:14.387762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:14.388373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:14.388654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:14.388930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:14.389652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] RBAC: Adding query filter params (role.id=47f258b925fe4809989a40c4b5c9472a, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=3c56f73fdd7742d6a336f41ff9da3578) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:14.389962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:14.391431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06832509-7d33-4664-ad0e-9d02a8ce3695 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:14.402056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 486/989] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:14 2022] GET /identity/v3/role_assignments?user.id=3c56f73fdd7742d6a336f41ff9da3578&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=47f258b925fe4809989a40c4b5c9472a => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:15.412983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149422b-b96c-4cd9-8d61-f1a1a9c1d7ff None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.413495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149422b-b96c-4cd9-8d61-f1a1a9c1d7ff None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.413571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149422b-b96c-4cd9-8d61-f1a1a9c1d7ff None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.415039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 504/990] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:15.419980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.420379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.420590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.438641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:15.439830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:15.466799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925ec4-7eca-4f1d-a4d2-8eb81dbb1190 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.467117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925ec4-7eca-4f1d-a4d2-8eb81dbb1190 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.467346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925ec4-7eca-4f1d-a4d2-8eb81dbb1190 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.468678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 505/991] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:15.473398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.473679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.473949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.492046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:15.493219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:15.518023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 487/992] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:15 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:15.561649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 506/993] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:15 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:15.782661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:15.823383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:15.843244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:15.886159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd8eee-f6fe-4a70-9115-3f0511220492 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:15.888633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:15.922305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:15.922940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.923219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.923489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.924070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:15.925439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:15.934259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:15.935287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:15.936139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:15.936969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ef46d4-8bcb-4e9f-ab32-cfa6fccf1e04 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:15.940315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 488/994] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:15 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:15.941155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:15.973042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:15.973676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:15.973952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:15.974292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:15.974878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:15.976236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:15.985489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:15.986538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:15.987391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 507/995] 10.222.0.46 () {56 vars in 1126 bytes} [Fri Oct 28 10:49:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:15.992851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.047455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39429519-d67f-4a28-80bf-0b631117675c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.074477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:16.082778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.083387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.084247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.084519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.085141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:16.085439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.086858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.091729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 489/996] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:15 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:16.096997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.098820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558ff561-1742-4b0f-a706-4a8188437f45 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.133410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:16.137730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.138376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.138667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.138942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.139577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:16.139893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.141257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.146789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 508/997] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:16.152776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.211328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574dc03d-7ad7-4b1e-b7e6-c3b157f26d6c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.259080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.259696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.259974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.260159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c07a0e7-f8b4-4e26-82af-cdaa8820f9a3 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.260286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.260876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:16.261180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.263150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.269268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 490/998] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:16 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:16.273629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.303047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.303889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.304491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.304778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.305556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:16.305858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.307242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.312284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 509/999] 10.222.0.46 () {56 vars in 1122 bytes} [Fri Oct 28 10:49:16 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:16.317903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.327607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:16.383345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:16.390082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84eb92-81a8-4732-a145-7d5da285a687 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.425475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2993df59-8f17-414c-8c3f-d8c16cb1d3e2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.428947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.429631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.429965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.430265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.430983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] RBAC: Adding query filter params (interface=public, service_id=328e1021e2b944fa8b832a12c85cc73a, region_id=RegionOne) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:16.431290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.432674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.437807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 491/1000] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:49:16 2022] GET /identity/v3/endpoints?service_id=328e1021e2b944fa8b832a12c85cc73a&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:16.444026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.466181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.466828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.467113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.467387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.468075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.469508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8856c71-8f97-427a-a1d8-9fcb0a75966e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.498005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 510/1001] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:16 2022] POST /identity/v3/endpoints => generated 325 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:49:16.560147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc21159-2f73-4fe1-ade5-df8a0855698e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:16.579214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e1eb47-b324-429c-9f54-4cbcb851efb4 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.579396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e1eb47-b324-429c-9f54-4cbcb851efb4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.579660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e1eb47-b324-429c-9f54-4cbcb851efb4 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.581227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 511/1002] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:16.587200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.587535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.587769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.603163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:16.603784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:16.604060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:16.604325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:16.604925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:16.606605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:16.606949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:16.608163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:16.622026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 492/1003] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:16 2022] GET /identity/v3/services => generated 1727 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 10:49:16.687644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:16.696554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 512/1004] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:16 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:16.963232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:16.986686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:17.066140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1cb73-6e1a-454a-8980-eef0e284d73c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:17.100165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:17.100793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:17.101069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:17.101338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] PATH_INFO: `/v3/services/image` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:17.101914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:17.103292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:17.112319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:17.113275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:17.114162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:17.118074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 493/1005] 10.222.0.46 () {56 vars in 1105 bytes} [Fri Oct 28 10:49:16 2022] GET /identity/v3/services/image => generated 86 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:17.119410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:17.220200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aba7a27-031c-49fa-b2bb-073c85c4cb9f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:17.259058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:17.259677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:17.259954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:17.260224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:17.260858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] RBAC: Adding query filter params (name=image) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:17.261155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:17.262549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2530b44-1e76-4550-9049-3d9bbfc7604c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:17.272450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 513/1006] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:17.274346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:17.362688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:17.381171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab2610-47e5-49d6-aa00-9b79726f82c0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:17.417492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:17.418133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:17.418414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:17.418685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:17.419312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] RBAC: Adding query filter params (type=image) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:17.419614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:17.420976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56984dfc-a708-4b33-95d6-d8d7d59ac916 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:17.434273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 494/1007] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:17 2022] GET /identity/v3/services?type=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:18.664502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9822f8-44ed-43b2-92a0-9730bb9e0caf None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:18.665048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9822f8-44ed-43b2-92a0-9730bb9e0caf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:18.665048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9822f8-44ed-43b2-92a0-9730bb9e0caf None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:18.666738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 514/1008] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:18.672431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cc736-0bfd-4419-9953-ffd35e54778e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:18.672932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cc736-0bfd-4419-9953-ffd35e54778e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:18.673005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cc736-0bfd-4419-9953-ffd35e54778e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:18.691996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-dc5cc736-0bfd-4419-9953-ffd35e54778e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:18.693750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-dc5cc736-0bfd-4419-9953-ffd35e54778e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:18.761122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 495/1009] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:18 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:18.787937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331fc58-4027-4875-9ffd-8c19be0dde75 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:18.788215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331fc58-4027-4875-9ffd-8c19be0dde75 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:18.788682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b331fc58-4027-4875-9ffd-8c19be0dde75 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:18.790024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 515/1010] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:18.795268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:18.795557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:18.795828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:18.814153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:18.815316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:18.879972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 496/1011] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:18 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:19.026073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b331fc58-4027-4875-9ffd-8c19be0dde75 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.086496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:19.129662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b331fc58-4027-4875-9ffd-8c19be0dde75 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.151591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.161410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:19.167049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.167975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.168254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.168528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.169628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.171101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.172661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 516/1012] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:19.178428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.268350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c969b3a-5009-4c9b-9457-2e62ceef3259 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.282621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ce6f9-be81-400d-b859-a661803c5140 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.312512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:19.312846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.313839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.313839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.314078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.314688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.316164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.322289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.322895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.323169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.323444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.324089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:19.325326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:19.326293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 10:49:19.327138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:19.327869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.331714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 497/1013] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:19 2022] GET /identity/v3/services/compute => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:19.332356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.366738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 517/1014] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:49:19 2022] POST /identity/v3/projects => generated 311 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:49:19.416730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:19.452658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ad2340-917d-4039-961c-8b49fbb2882e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.494215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.494873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.495137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.495416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.496075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:19.496420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.497811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d948b070-7a9b-41eb-a5cb-cb0160fb6528 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.508618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 498/1015] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:19 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:19.510329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.587779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ed192-a096-4c4d-a166-f6bbf027a703 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.588068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ed192-a096-4c4d-a166-f6bbf027a703 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.588349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214ed192-a096-4c4d-a166-f6bbf027a703 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.589694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 499/1016] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:19.595482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.596387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.596851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.616966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:19.618250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:19.621363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119257e-5a25-4a23-8d3c-4f5811c49a75 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.657697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.658343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.658620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.659308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.659949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:19.660254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.662596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.667980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 518/1017] 10.222.0.46 () {56 vars in 1112 bytes} [Fri Oct 28 10:49:19 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:19.674267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:19.689109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 500/1018] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:19 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:19.724440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:19.782197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df27c637-d9cc-4860-846d-eb0e4787d217 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:19.815658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:19.816272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:19.816549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:19.816819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:19.817533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:19.819453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0dc45e-fc8b-400c-a217-f096c22097a0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:19.846099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 519/1019] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:19 2022] POST /identity/v3/endpoints => generated 334 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:49:19.960237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:20.032592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:20.069492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75eebf6-b350-4b9a-aeb8-454b23a7a942 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:20.116027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:20.116569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:20.116848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:20.117120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:20.117775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:20.119186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c7864e-cd61-4d77-9a61-57e078434f11 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:20.138783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 501/1020] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:49:19 2022] POST /identity/v3/services => generated 242 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:21.570356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8823b44-0e21-433e-b1b4-0b034d5349af None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:21.570884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8823b44-0e21-433e-b1b4-0b034d5349af None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:21.570884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8823b44-0e21-433e-b1b4-0b034d5349af None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:21.572290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 520/1021] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:21.577298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:21.577684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:21.577842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:21.595772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:21.596949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:21.670255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 502/1022] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:21 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:21.938310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8823b44-0e21-433e-b1b4-0b034d5349af None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.044439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.048938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8823b44-0e21-433e-b1b4-0b034d5349af None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.084642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.085269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.085549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.085816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.086948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:22.088374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ca16d1-199e-4cd7-802d-0494f01bd85d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:22.090020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 521/1023] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:22.095822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.123750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3840560d-624a-4990-af6e-6093591c3134 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.124301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3840560d-624a-4990-af6e-6093591c3134 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.124733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3840560d-624a-4990-af6e-6093591c3134 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.126508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 522/1024] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:22.133163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a632cfa-c77f-43b0-9ffd-fffc3211bfdd None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.133449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a632cfa-c77f-43b0-9ffd-fffc3211bfdd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.133723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a632cfa-c77f-43b0-9ffd-fffc3211bfdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.147444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.153768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9a632cfa-c77f-43b0-9ffd-fffc3211bfdd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:22.154991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9a632cfa-c77f-43b0-9ffd-fffc3211bfdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:22.206600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45231750-90c0-4cf1-8ba4-056bf0fd70e9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.231450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 523/1025] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:22 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:22.252161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.252789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.253067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.253336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.254031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:22.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9414ef-d7f1-4047-8eb1-0d381093bec2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:22.280197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 503/1026] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:49:22 2022] POST /identity/v3/users => generated 285 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:22.329028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57248b70-6bc7-417d-a4c7-6a1d3448c2e0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.329399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57248b70-6bc7-417d-a4c7-6a1d3448c2e0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.329599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57248b70-6bc7-417d-a4c7-6a1d3448c2e0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.331230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 524/1027] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:22.337505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.337813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.338111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.358697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:22.359927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:22.434907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 504/1028] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:22 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:22.525578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57248b70-6bc7-417d-a4c7-6a1d3448c2e0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.621225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.627379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57248b70-6bc7-417d-a4c7-6a1d3448c2e0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.661894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.662555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.662835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.663107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.673329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:22.678182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:22.679093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:22.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 525/1029] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:22.685928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.707423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.721643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.794377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944fc68c-0aa4-45b4-a3a1-d3fa892bf29f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.822263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074a030b-c5bb-438c-aede-1e6a4fcd85b2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.829854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.830508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.830785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.831052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.831715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:22.832016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:22.833371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:22.842065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 505/1030] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:22.847830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.862575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.863206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.863484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.863750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] PATH_INFO: `/v3/services/image` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:22.864321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:22.866295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:22.875175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:22.876262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:22.877129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 526/1031] 10.222.0.46 () {56 vars in 1105 bytes} [Fri Oct 28 10:49:22 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:22.882672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:22.908375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.955038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:22.962264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678dff6-7e70-4112-95fe-37b3db1bb193 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.991766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dce612-5349-44cb-808c-994f9c1cb2ef None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:22.997283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:22.997919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:22.998222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:22.998489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.010755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.012209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:23.029225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:23.030226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:23.031123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:23.031123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:23.031123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:23.031294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.031294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.031294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.031294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.031573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] RBAC: Adding query filter params (name=image) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:23.031796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.038107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.038233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 506/1032] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:23.038782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.042418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 527/1033] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:22 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:23.047727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.142689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.147249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c06252-783f-424b-ae38-91322e0d804c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.159167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1a9987-7067-466e-856e-019c369a19cd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.181177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.181787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.182097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.182372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.183004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:23.183303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.184668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.194582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.195195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.195477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.195751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.196378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] RBAC: Adding query filter params (type=image) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:23.196681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.197816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 507/1034] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:23.202851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.203601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.207703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 528/1035] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/services?type=image => generated 350 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:23.209145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.215783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.320822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f5de32-c599-4bc8-8f42-0e7c9300dc25 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.321213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8f66c-703c-4d33-9b26-6e6f0124cca1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.356578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.357180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.357457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.357725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.358442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] RBAC: Adding query filter params (interface=public, service_id=0221fdb059e14457aa67f178bbbee706, region_id=RegionOne) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:23.358749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.359133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.359768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.360051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.360147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.360325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.365306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 529/1036] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/endpoints?service_id=0221fdb059e14457aa67f178bbbee706&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:23.367682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.369111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.370906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:23.381843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:23.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:23.383852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:23.384387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 508/1037] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:23.388388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.389877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.460340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.484534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe1d456-4bd4-4ce5-9578-882cf2d80ffa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.497694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128e49d-4319-4d57-a618-5d6aab4b4221 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.523467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.524136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.524424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.524702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.525313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.526701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.532829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.533174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 530/1038] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/services => generated 1957 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 10:49:23.533825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.534205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.534481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.535201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:23.535506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.537047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.560826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 509/1039] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:23.566916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:23.656444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.682134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646061c9-2060-435e-9d5d-910fc353f7dd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:23.722056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:23.722677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:23.722956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:23.723232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:23.792886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:23.794362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:23.870333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:23.898659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 531/1040] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:23.899886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:24.001740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f632f0-70ac-487e-906c-2eb92a3a1a3a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:24.029830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:24.040214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:24.040847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.041126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.041392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.051382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:24.053407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:24.068136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:24.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:24.070511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:24.070511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:24.070511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:24.070511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 510/1041] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:24.074962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:24.172770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11743b8b-447c-4d07-baaf-915560b2d3aa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:24.206380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:24.206985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.207261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.207531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.208164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:24.208489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:24.209841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:24.226018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 532/1042] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:24.227763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:24.337780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40d30e-a9fb-400f-8323-0c468a77bd85 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:24.382482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:24.383117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.383396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.383746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.390460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:24.391895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1f1f1b7a-b3c4-48f7-b4ce-78b686210072 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:24.404353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:24.405325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:24.406276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:24.407349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 511/1043] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:24.412417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:24.459244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:24.524062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd692de9-dc81-4d70-886b-7bc1889d4cc5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:24.534219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521af5c-e9e5-42a2-83f7-c14c4c01856d None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.534503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521af5c-e9e5-42a2-83f7-c14c4c01856d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.535115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b521af5c-e9e5-42a2-83f7-c14c4c01856d None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.536518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 512/1044] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:24.542415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117276e8-a077-494b-8179-28f468c7741c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.542695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117276e8-a077-494b-8179-28f468c7741c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.542964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117276e8-a077-494b-8179-28f468c7741c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.559382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:24.560035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.560320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.560588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.561263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:24.561564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:24.561661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-117276e8-a077-494b-8179-28f468c7741c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:24.562854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-117276e8-a077-494b-8179-28f468c7741c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:24.563026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:24.579312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 533/1045] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:24.585921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:24.595430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:24.636049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 513/1046] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:24 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:24.687482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:24.691879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421013df-c346-4c03-a519-759ff4d96bfa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:24.726320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:24.726944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:24.727220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:24.727487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:24.728199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] RBAC: Adding query filter params (role.id=47f258b925fe4809989a40c4b5c9472a, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:24.728503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:24.729945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:24.742230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 534/1047] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:24 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=47f258b925fe4809989a40c4b5c9472a => generated 259 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:24.908137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117276e8-a077-494b-8179-28f468c7741c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.006211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117276e8-a077-494b-8179-28f468c7741c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:25.039906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:25.040536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.040810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.041074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.051089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:25.055887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:25.056818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:25.057651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 514/1048] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:49:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:25.063102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.166185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7785f634-6dce-4184-a4c9-4ac29f46f0d5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:25.200879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:25.201495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.201776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.202076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.202737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:25.203039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:25.204395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:25.214875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 535/1049] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:25.220529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.255016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:25.331294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f126a-a4b3-4c73-b0c7-a002a44c9eaf None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:25.366569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:25.367201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.367475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.367741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] PATH_INFO: `/v3/users/197ab7faf50e47b0abfb89c901dc7d1b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.380915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] RBAC: Authorizing `identity:get_user(user_id=197ab7faf50e47b0abfb89c901dc7d1b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:25.382371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabe587d-f0af-4a69-9bec-8c1b53982117 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:25.390251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 515/1050] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:25 2022] GET /identity/v3/users/197ab7faf50e47b0abfb89c901dc7d1b => generated 285 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:25.390397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.420087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:25.498143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd32b2-de9a-4696-831a-6f337b0223ce None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:25.531754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:25.532367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.532648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.532920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.604135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:25.605949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:25.632072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:25.710101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 536/1051] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:25 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:25.712382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04881a0d-1d97-4989-bbd3-72bdf9407958 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.712713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.712934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04881a0d-1d97-4989-bbd3-72bdf9407958 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.713097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04881a0d-1d97-4989-bbd3-72bdf9407958 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.714571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 516/1052] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:25.719574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.719856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.720127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.738384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:25.739575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:25.807323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 517/1053] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:25 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:25.821547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edce066b-de99-4cbf-9fe6-006906fd38ec None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:25.846070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:25.857894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:25.858552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:25.858839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:25.859098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:25.860216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:25.861631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:25.863432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 537/1054] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:25 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:25.869321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:25.887853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:25.971116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02a08a-715e-4bd3-aa7a-a60fb001a9db None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:26.004539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:26.005143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.006198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.006368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.006964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=1beffc1ef50341298191dcf3e1d4f29e, user.id=197ab7faf50e47b0abfb89c901dc7d1b) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:26.007265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:26.008683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:26.014253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 518/1055] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:25 2022] GET /identity/v3/role_assignments?user.id=197ab7faf50e47b0abfb89c901dc7d1b&scope.project.id=1beffc1ef50341298191dcf3e1d4f29e&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:26.078273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:26.187133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df088b88-dbf4-4f56-bef5-85796f13b169 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:26.227768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:26.228325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.228600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.228867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] PATH_INFO: `/v3/services/image` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.229438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:26.231071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:26.239809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:26.240779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:26.241628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 538/1056] 10.222.0.46 () {56 vars in 1105 bytes} [Fri Oct 28 10:49:26 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:26.248131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:26.337009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:26.357981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3f589f-55f7-4bf3-af3f-8b666bda80a3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:26.391549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:26.392161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.392441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.392711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.393334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] RBAC: Adding query filter params (name=image) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:26.393635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:26.395029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:26.405451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 519/1057] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:26.406420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:26.463299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:26.509275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48905606-625b-49c7-8861-5034e6a09a26 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:26.542860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:26.543470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.543748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.544017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.544641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] RBAC: Adding query filter params (type=image) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:26.544940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:26.546335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452486b1-0bec-4854-b14f-8d63c7a7d538 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:26.555772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 539/1058] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:26 2022] GET /identity/v3/services?type=image => generated 350 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:26.557001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:26.657097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9c2b7-f56b-43d1-a237-d991ba51b401 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:26.690654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:26.691269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.691546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.691814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.692488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:26.694456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb126f9e-8f2f-49e0-b9f8-902bea58cd13 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:26.710503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:26.726006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 520/1059] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:49:26 2022] POST /identity/v3/endpoints => generated 327 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:49:26.879917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d216b5e2-787b-42f1-8277-4e2f5ad457a8 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.880546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d216b5e2-787b-42f1-8277-4e2f5ad457a8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.880546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d216b5e2-787b-42f1-8277-4e2f5ad457a8 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.882132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 540/1060] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:26.887611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaa9bf-a194-402f-896d-62811359342f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:26.887917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaa9bf-a194-402f-896d-62811359342f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:26.888172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaa9bf-a194-402f-896d-62811359342f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:26.907625 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-49eaa9bf-a194-402f-896d-62811359342f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:26.908808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-49eaa9bf-a194-402f-896d-62811359342f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:26.983713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 521/1061] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:27.259847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d216b5e2-787b-42f1-8277-4e2f5ad457a8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:27.371300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d216b5e2-787b-42f1-8277-4e2f5ad457a8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:27.391912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:27.407801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:27.408450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:27.408727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:27.409176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:27.421455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:27.426633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:27.427773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:27.428609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:27.435990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 541/1062] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:27.436952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaa9bf-a194-402f-896d-62811359342f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:27.549690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaa9bf-a194-402f-896d-62811359342f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:27.576171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:27.585811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:27.586451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:27.586725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:27.586995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:27.587654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:27.587954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:27.589311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:27.602019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 522/1063] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:27.603474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:27.709486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de2339-8692-4a2b-beb6-fc1a9df8ce43 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:27.743522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:27.744148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:27.744423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:27.744690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:27.755140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:27.757105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:27.771374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:27.772322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:27.773209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:27.773209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:27.773209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:27.773209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 542/1064] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:27.779176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:27.879982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bfda7f-da6f-4e12-af17-1b2556d4efd1 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:27.913052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:27.913739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:27.914077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:27.914342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:27.914996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:27.915294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:27.916767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:27.929409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 523/1065] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:27.934991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:27.968135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.042540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff12e21-4337-4418-a093-1cb320fd19a2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.076714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.077338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.077613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.077903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.085836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.087286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-e8d52271-c0f8-49f7-b8a7-aea5fb0d1fe1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:28.098165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:28.099148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:28.100077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 543/1066] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:49:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:28.106986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.127778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764b9086-72d6-407d-ac91-2c118664000a None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.128059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764b9086-72d6-407d-ac91-2c118664000a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.128321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764b9086-72d6-407d-ac91-2c118664000a None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.129640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 544/1067] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:49:28.135197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.135415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.135682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.155280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:28.156439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:28.158368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.216015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46263b56-0860-4d5a-8839-30a9291cb0b3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.228900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 545/1068] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:49:28.256317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.256928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.257204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.257467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.258216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:28.258517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.259928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.280950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 524/1069] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:28.286640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.310368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.387554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1683ef-83c5-4d97-afff-72645c44d173 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.422844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.423472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.423804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.424079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.434789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.436224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:28.451156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:28.452112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:28.457763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:28.457763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:28.457763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:28.457763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 546/1070] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:28.459764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.518084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.560239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2f4891-e442-44ae-a9d0-a1b435c54d3e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.569056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.601315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.603460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.604479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.604823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.605630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.606476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:28.606873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.608130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.620597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 525/1071] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:28.626215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.629097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac9e59a-e441-4d2d-8be6-b5f8b74c0994 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.665069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.665648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.665941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.666233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.677870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:28.682720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:28.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:28.684482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 547/1072] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:28.690038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.735930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30f9455-3048-43f1-9caa-657bc5742ae9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.781183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.781821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.782135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.782407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.792205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.794371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.795339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1de6a1-4064-4fb4-9197-41ee9d210615 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:28.806825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:28.807931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:28.808969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:28.810809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 526/1073] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:28.817013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.839610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.840574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.841044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.841622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.842346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:28.842657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.844055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.845788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.848283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:28.858794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 548/1074] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:28.860017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.933472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6808e2-bada-45d2-99b2-0cd139fe0e7c None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.964264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5443d3ae-8d73-48e5-af60-a4b29ca539ba None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:28.966495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.967098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.967375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.967644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.968328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:28.968632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:28.970083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109b77aa-8b04-4d7e-a525-16b5e4e2c952 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:28.982838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 527/1075] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:28.984254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6144b-3b4b-4d4d-b26e-941a614467e3 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.984391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6144b-3b4b-4d4d-b26e-941a614467e3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:28.984613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6144b-3b4b-4d4d-b26e-941a614467e3 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:28.985794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 528/1076] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:28.988570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6144b-3b4b-4d4d-b26e-941a614467e3 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:28.999015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:28.999652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:28.999930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.000200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] PATH_INFO: `/v3/users/197ab7faf50e47b0abfb89c901dc7d1b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.001336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] RBAC: Authorizing `identity:get_user(user_id=197ab7faf50e47b0abfb89c901dc7d1b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.002789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cad70-9ec5-47f9-b447-561732e54531 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.004362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 549/1077] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:28 2022] GET /identity/v3/users/197ab7faf50e47b0abfb89c901dc7d1b => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:29.006113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.006395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.006668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.023886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:29.025061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:29.073103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:29.082405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:29.089011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6144b-3b4b-4d4d-b26e-941a614467e3 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:29.092048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 550/1078] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:29.092905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:29.132682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:29.134634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.134917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.135190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/47f258b925fe4809989a40c4b5c9472a` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.137313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=6adb6b4eff2545c080d2eb3327b78d0f, role_id=47f258b925fe4809989a40c4b5c9472a)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.138911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.161956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 529/1079] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:49:28 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/6adb6b4eff2545c080d2eb3327b78d0f/roles/47f258b925fe4809989a40c4b5c9472a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:49:29.201879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0236e7d3-d109-43c8-8a96-cf5a1b5f1cde None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:29.238569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:29.239222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.240064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.240344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.241495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.242971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.244678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 551/1080] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:29 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:29.250669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:29.362872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:29.371103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c516d68c-f0f0-45b2-81ea-e4a36d84e989 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:29.408611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:29.409269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.409547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.409998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e/users/197ab7faf50e47b0abfb89c901dc7d1b/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.412218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] RBAC: Authorizing `identity:create_grant(project_id=1beffc1ef50341298191dcf3e1d4f29e, user_id=197ab7faf50e47b0abfb89c901dc7d1b, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.413660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.428528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 530/1081] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:49:29 2022] PUT /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e/users/197ab7faf50e47b0abfb89c901dc7d1b/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:49:29.439372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:29.485698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19c99-14ec-4267-9fd9-691a849b919c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:29.524242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:29.525151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.525574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.526033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.601910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.603399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.665057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:29.704506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 552/1082] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:29.710906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:29.719575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:29.812470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d87396-8f8d-4296-bc5b-9f6f601fbb00 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:29.845848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:29.846513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:29.846787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:29.847058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:29.858479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:29.859929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-a875bf68-f491-4055-ae29-8310f86522fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:29.874278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:29.875262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:29.876721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:29.876721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:29.876721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:29.876721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 531/1083] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:29.881329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:29.981168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bf56f4-240d-468e-a127-6a56b3d775cf None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:30.014458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:30.015063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:30.015339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:30.015609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:30.016238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:30.016536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:30.017885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e239511b-8834-4066-b603-a50b4d13f7f4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:30.038004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 553/1084] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:31.326943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62805f3e-563c-46d8-8861-63fb4de4d777 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.327461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62805f3e-563c-46d8-8861-63fb4de4d777 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.327461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62805f3e-563c-46d8-8861-63fb4de4d777 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.328980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 532/1085] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:31.334488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4536d2e-cc92-44cd-8c12-e90185b783cc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.334872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4536d2e-cc92-44cd-8c12-e90185b783cc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.335043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4536d2e-cc92-44cd-8c12-e90185b783cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.353495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b4536d2e-cc92-44cd-8c12-e90185b783cc None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:31.354734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b4536d2e-cc92-44cd-8c12-e90185b783cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:31.370975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:31.420559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 554/1086] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:31.584781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48abbd-9751-4238-b029-237b4e575096 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.585073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48abbd-9751-4238-b029-237b4e575096 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.585349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48abbd-9751-4238-b029-237b4e575096 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.586711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 533/1087] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:31.591837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.592205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.592490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.610577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:31.611760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:31.683763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 555/1088] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:31.692516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc48abbd-9751-4238-b029-237b4e575096 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:31.750986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:31.794323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc48abbd-9751-4238-b029-237b4e575096 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:31.826881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:31.827496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.827771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.828039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.838931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:31.843645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:31.844781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:31.845785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 534/1089] 10.222.0.46 () {56 vars in 1120 bytes} [Fri Oct 28 10:49:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:31.851380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:31.929855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:31.948027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de2792-9986-45e6-b412-e25e05292e1b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:31.951129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:31.982884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:31.983487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:31.983759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:31.984025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:31.984680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:31.984978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:31.986357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:31.994158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 556/1090] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:49:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:31.999866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.054559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f60461-64ee-4c4e-9a05-0b83ecf2f897 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.070970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.088811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.089488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.089718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.090015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.096729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6635848-6010-4753-9548-7d01a2294166 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.100401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:32.105872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:32.106783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:32.107759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 535/1091] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:49:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:32.112886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.130529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.131159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.131434 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.131702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.141795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.143277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.146272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:32.159105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:32.160065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:32.160951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:32.160951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:32.160951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:32.160951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 557/1092] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:32.166490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.212597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91264fc0-2ebe-49dc-ad93-bda5f7ae2df9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.245230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.245830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.246133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.246402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.247059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:32.247358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.248756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.257471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 536/1093] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:32.263129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.264697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d79892-1371-4c68-9f8a-f09b24ac0d5b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.300543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.301151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.301434 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.301707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.302407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:32.302709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.304065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.306747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.315984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 558/1094] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:32.322240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.364540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a14cb3-7dfd-43e1-913f-d0114294913d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.377521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.398926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.399556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.399838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.400112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] PATH_INFO: `/v3/users/197ab7faf50e47b0abfb89c901dc7d1b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.401248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] RBAC: Authorizing `identity:get_user(user_id=197ab7faf50e47b0abfb89c901dc7d1b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.402704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.404346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 537/1095] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/users/197ab7faf50e47b0abfb89c901dc7d1b => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:32.409890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.422616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819627f7-9cfc-467b-bf85-3a8a572af63a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.456878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.457504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.457788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.458091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.465845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.467309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:49:32.479096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:32.480239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 10:49:32.481862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:32.485258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 559/1096] 10.222.0.46 () {56 vars in 1093 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:32.486115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.508315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce162c48-5090-45b4-998a-9768766d6f98 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.517260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.542876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.543477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.543753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.544022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.586065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cce7271-885e-4b22-b1d7-098ead4a636b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.601403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.611331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.612797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.619471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.620013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.620292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.620562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.621278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:32.621578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.623004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.640068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 560/1097] 10.222.0.46 () {56 vars in 1133 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:32.645582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.710305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 538/1098] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 10:49:32.716220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.725437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.743201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fb52bf-fc0d-4f84-9c25-a189a20dc101 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.777655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.778294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.778574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.778848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.814697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593530b4-29c0-44e3-accb-e6f81f0cc0e9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.821234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.845763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.847308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.847974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.848599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.848909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.849190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.850324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.851726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:32.853486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 539/1099] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:32.859707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.935257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:32.944547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 561/1100] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 10:49:32.950202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:32.960363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475c903d-25a1-4034-8b11-7c8f72638a4e None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:32.994217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:32.994818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:32.995097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:32.995370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:32.996090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=1beffc1ef50341298191dcf3e1d4f29e, user.id=197ab7faf50e47b0abfb89c901dc7d1b) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:32.996400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:32.997837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.003524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 540/1101] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/role_assignments?user.id=197ab7faf50e47b0abfb89c901dc7d1b&scope.project.id=1beffc1ef50341298191dcf3e1d4f29e&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:33.023731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:33.051467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ebddcc-b5a2-4947-a063-0b1ebbd86cb1 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:33.088847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:33.089489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:33.089764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:33.090060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:33.101772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:33.103269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:49:33.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:33.120625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:49:33.121510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:49:33.121510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:49:33.121510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:33.121510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 562/1102] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:49:33.126538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:33.230247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6eb92-b961-4523-826b-3e2d17cbb7d2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:33.263724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:33.264332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:33.264607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:33.264877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:33.265513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:33.265815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:33.267241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.276919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:33.280453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 541/1103] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:49:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:33.286052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:33.383825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426b2fb-b17e-42f6-9e85-0336bddd3a56 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:33.408172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:33.416871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:33.417500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:33.417778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:33.418077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:33.424535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:33.425965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:49:33.437479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:33.438404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:49:33.439279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 563/1104] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:49:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:33.445105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:33.541032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787ace5c-e9e3-4bd3-98ae-83064e8d110f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:33.574451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:33.575045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:33.575320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:33.575588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:33.576263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:33.576562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:33.577985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7ff8e4-b1e2-400e-84c1-74eb87413f7b None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.589164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 542/1105] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:49:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:33.595173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:33.691312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98bc6a-7ec9-4eb0-bf90-869c865ca675 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:33.724741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:33.725347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:33.725635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:33.725904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:33.726654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] RBAC: Adding query filter params (role.id=47f258b925fe4809989a40c4b5c9472a, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=6adb6b4eff2545c080d2eb3327b78d0f) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:33.726962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:33.728380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea58e8-5f4b-4213-8666-415fb0661d64 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:33.734556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 564/1106] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:33 2022] GET /identity/v3/role_assignments?user.id=6adb6b4eff2545c080d2eb3327b78d0f&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=47f258b925fe4809989a40c4b5c9472a => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:35.191350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba535ba6-0e2a-4522-b675-58f8cbc6b38f None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:35.191881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba535ba6-0e2a-4522-b675-58f8cbc6b38f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:35.191881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba535ba6-0e2a-4522-b675-58f8cbc6b38f None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:35.193218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 543/1107] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:35.198339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:35.198753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:35.198890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:35.217086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:35.218300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:35.262977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:35.283842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 565/1108] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:35.549051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba535ba6-0e2a-4522-b675-58f8cbc6b38f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:35.607470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:35.648828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba535ba6-0e2a-4522-b675-58f8cbc6b38f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:35.681614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:35.682245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:35.682521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:35.682789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:35.683867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:35.685277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104f5886-5c1e-4578-8738-a4872bd13f1d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:35.686811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 544/1109] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:35.692383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:35.787111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354519f-0baf-4a2d-bd61-b70d969f5124 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:35.820069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:35.820601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:35.820877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:35.821148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:35.821787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:35.823236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136619d6-ca51-4ae6-8c83-eed51eccc990 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:35.843664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 566/1110] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:49:35 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:37.973373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7887c-28f1-4d0b-a559-add285107e1e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:37.976048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7887c-28f1-4d0b-a559-add285107e1e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:37.976048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7887c-28f1-4d0b-a559-add285107e1e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:37.976048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 545/1111] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:37.981096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:37.981511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:37.981636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:37.999692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:38.000853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:38.028444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:38.066718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 567/1112] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:38.328110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7887c-28f1-4d0b-a559-add285107e1e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:38.428633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7887c-28f1-4d0b-a559-add285107e1e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:38.453117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:38.460889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:38.461510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:38.461784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:38.462082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] PATH_INFO: `/v3/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:38.463439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] RBAC: Authorizing `identity:get_role(role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:38.464798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:38.466109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 546/1113] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:38 2022] GET /identity/v3/roles/1126c16f80da4dde8de7192a7858171e => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:38.472247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:38.568349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a90820-6c36-488c-9815-b290e97f3b22 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:38.600590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:38.601209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:38.601487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:38.601758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] PATH_INFO: `/v3/users/9cdd867095984edc86e36cf96d87eb49` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:38.615247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] RBAC: Authorizing `identity:get_user(user_id=9cdd867095984edc86e36cf96d87eb49)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:38.616663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:38.618257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 568/1114] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:38 2022] GET /identity/v3/users/9cdd867095984edc86e36cf96d87eb49 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:38.623941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:38.718710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632037fe-1777-44b2-be34-88c51f8550c8 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:38.750707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:38.751322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:38.751595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:38.751860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:38.809705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:38.816493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:38.818592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:38.910339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 547/1115] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:38.916586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:38.941178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:39.013305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf5a49c-0eda-4510-86ee-d270ed6a9e77 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:39.045665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:39.046313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:39.046589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:39.046858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:39.047950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:39.049359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900f9db3-4c5d-4d8f-8fae-a3ed26c50e0d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:39.051015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 569/1116] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:38 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:39.057730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:39.147985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:39.154090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc2240-269d-43cd-b259-27c9b0ac4a66 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:39.187588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:39.188171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:39.188447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:39.188714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:39.189424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=1beffc1ef50341298191dcf3e1d4f29e, user.id=9cdd867095984edc86e36cf96d87eb49) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:39.189727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:39.191160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979f3b3-57e9-45f7-afd5-fbb714efe88c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:39.196342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 548/1117] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:39 2022] GET /identity/v3/role_assignments?user.id=9cdd867095984edc86e36cf96d87eb49&scope.project.id=1beffc1ef50341298191dcf3e1d4f29e&role.id=1126c16f80da4dde8de7192a7858171e => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:41.281183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6162cb6f-b932-419e-a856-a2fc6db6bb64 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:41.281596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6162cb6f-b932-419e-a856-a2fc6db6bb64 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:41.281725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6162cb6f-b932-419e-a856-a2fc6db6bb64 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:41.283088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 570/1118] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:41.288644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:41.289009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:41.289190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:41.306945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:41.308126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:41.372261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 549/1119] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:41.638049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6162cb6f-b932-419e-a856-a2fc6db6bb64 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:41.725566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:41.735492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6162cb6f-b932-419e-a856-a2fc6db6bb64 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:41.768767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:41.769389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:41.769666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:41.769938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] PATH_INFO: `/v3/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:41.771412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] RBAC: Authorizing `identity:get_role(role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:41.772788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:41.774097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 571/1120] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:41 2022] GET /identity/v3/roles/1126c16f80da4dde8de7192a7858171e => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:41.779689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:41.866152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:41.876062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73511b9-a409-4f98-9182-433d0110eecf None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:41.908815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:41.909390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:41.909673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:41.909949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] PATH_INFO: `/v3/users/9cdd867095984edc86e36cf96d87eb49` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:41.911095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] RBAC: Authorizing `identity:get_user(user_id=9cdd867095984edc86e36cf96d87eb49)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:41.912534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:41.914125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 550/1121] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:41 2022] GET /identity/v3/users/9cdd867095984edc86e36cf96d87eb49 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:41.919812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:42.015683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7999b765-77e0-47c6-b56c-ad0e89ad83ec None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:42.048018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:42.048628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:42.048904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:42.049174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:42.050292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:42.051732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e68f40-cbcc-47a6-ab70-bb7034858cee None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:42.054210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 572/1122] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:41 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:42.059961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:42.156431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60abd30f-03b8-4ddf-a639-c6659e9fd425 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:42.190338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:42.190930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:42.191212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:42.191487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e/users/9cdd867095984edc86e36cf96d87eb49/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:42.193658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] RBAC: Authorizing `identity:create_grant(project_id=1beffc1ef50341298191dcf3e1d4f29e, user_id=9cdd867095984edc86e36cf96d87eb49, role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:42.195126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a688ba6-5aba-4d87-9110-8c0d90aa4486 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:42.207549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 551/1123] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:49:42 2022] PUT /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e/users/9cdd867095984edc86e36cf96d87eb49/roles/1126c16f80da4dde8de7192a7858171e => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:49:44.286829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fd1a1-0759-4c3c-aee7-c66d9112e615 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:44.287299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fd1a1-0759-4c3c-aee7-c66d9112e615 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:44.287415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fd1a1-0759-4c3c-aee7-c66d9112e615 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:44.288732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 573/1124] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:44.293808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:44.294238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:44.294387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:44.312163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:44.313353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:44.326657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:44.379228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 552/1125] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:44.644171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323fd1a1-0759-4c3c-aee7-c66d9112e615 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:44.672141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:44.743442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323fd1a1-0759-4c3c-aee7-c66d9112e615 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:44.776052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:44.776669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:44.776950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:44.777225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] PATH_INFO: `/v3/roles/1126c16f80da4dde8de7192a7858171e` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:44.778641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] RBAC: Authorizing `identity:get_role(role_id=1126c16f80da4dde8de7192a7858171e)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:44.780002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:44.781297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 574/1126] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:44 2022] GET /identity/v3/roles/1126c16f80da4dde8de7192a7858171e => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:44.787220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:44.883312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a2e90d-4101-4035-ae53-6174179ddf50 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:44.915670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:44.916232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:44.916509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:44.916781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] PATH_INFO: `/v3/users/9cdd867095984edc86e36cf96d87eb49` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:44.917897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] RBAC: Authorizing `identity:get_user(user_id=9cdd867095984edc86e36cf96d87eb49)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:44.919356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:44.920897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 553/1127] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:44 2022] GET /identity/v3/users/9cdd867095984edc86e36cf96d87eb49 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:44.927197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:45.017753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:45.023734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf335a0-316d-4a28-b1c2-bcd560d47655 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:45.056893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:45.057491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:45.057773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:45.058083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:45.122566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:45.123991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:45.211992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:45.218745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 575/1128] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:45.225065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:45.243568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:45.323276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928bf279-c6b9-4f10-8114-cf389ee38384 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:45.355392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:45.356259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:45.356531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:45.356798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] PATH_INFO: `/v3/projects/1beffc1ef50341298191dcf3e1d4f29e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:45.357886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] RBAC: Authorizing `identity:get_project(project_id=1beffc1ef50341298191dcf3e1d4f29e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:45.359333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b91884-6eaa-4422-81b8-0524355471cd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:45.360917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 554/1129] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:45 2022] GET /identity/v3/projects/1beffc1ef50341298191dcf3e1d4f29e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:45.367613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:45.461680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6e726-0451-4cec-8ed8-7df42c4d6e67 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:45.494288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:45.494819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:45.495093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:45.495359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:45.496065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] RBAC: Adding query filter params (role.id=1126c16f80da4dde8de7192a7858171e, scope.project.id=1beffc1ef50341298191dcf3e1d4f29e, user.id=9cdd867095984edc86e36cf96d87eb49) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:45.496365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:45.497765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69816c28-1ceb-4211-927f-efb4c245c08a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:45.503264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 576/1130] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:45 2022] GET /identity/v3/role_assignments?user.id=9cdd867095984edc86e36cf96d87eb49&scope.project.id=1beffc1ef50341298191dcf3e1d4f29e&role.id=1126c16f80da4dde8de7192a7858171e => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:47.609696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc43c97-d13e-462d-aada-2de8f8df8854 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:47.610365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc43c97-d13e-462d-aada-2de8f8df8854 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:47.610365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc43c97-d13e-462d-aada-2de8f8df8854 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:47.611578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 555/1131] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:47.616671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:47.617076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:47.617221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:47.635431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:47.636605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:47.690078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:47.701629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 577/1132] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:47.965121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc43c97-d13e-462d-aada-2de8f8df8854 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:48.046940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:48.063725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc43c97-d13e-462d-aada-2de8f8df8854 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:48.095470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:48.096079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:48.096355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:48.096623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:48.097705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:48.099689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea37c28-fba3-4fcf-90bc-6280cbb623e7 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:48.101203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 556/1133] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:48.107274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:48.201459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aefa3ac-0229-4c48-867a-23d1529c9152 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:48.233909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:48.234466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:48.234745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:48.235019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:48.235665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:48.237074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85c69d9-d799-4f3c-afa3-2f32e0eb6aa9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:48.262946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 578/1134] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:49:48 2022] POST /identity/v3/projects => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:49:50.407977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8824b35-bedd-4495-be43-c9de2ad28f27 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:50.408518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8824b35-bedd-4495-be43-c9de2ad28f27 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:50.408518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8824b35-bedd-4495-be43-c9de2ad28f27 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:50.409853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 557/1135] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:50.415023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:50.415449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:50.415575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:50.433286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:50.434487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:50.469212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:50.509190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 579/1136] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:50.778628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8824b35-bedd-4495-be43-c9de2ad28f27 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:50.876861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8824b35-bedd-4495-be43-c9de2ad28f27 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:50.909622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:50.910270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:50.910552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:50.910857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:50.911957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:50.913375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5922f84a-29c6-4c4a-b4ac-7360d876f88d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:50.914929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 558/1137] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:49:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:50.921109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:51.016259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff1aa4-b46f-4766-9441-235452449e9a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:51.048305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:51.048843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:51.049116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:51.049380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:51.050040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:51.051475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743617c7-e913-4237-969a-7361db1747f9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:51.071942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 580/1138] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:49:50 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:49:53.217810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deae602-139c-4d1a-9fd2-182c87a7a0a6 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:53.218375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deae602-139c-4d1a-9fd2-182c87a7a0a6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:53.218375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3deae602-139c-4d1a-9fd2-182c87a7a0a6 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:53.219696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 559/1139] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:53.224865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:53.225284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:53.225414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:53.244548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:53.245730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:53.268383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:53.312117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 581/1140] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:53.577268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deae602-139c-4d1a-9fd2-182c87a7a0a6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:53.598985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:53.676195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3deae602-139c-4d1a-9fd2-182c87a7a0a6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:53.708377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:53.708986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:53.709261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:53.709526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:53.719072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:53.724355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:53.725493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:53.726621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 560/1141] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:49:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:53.731726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:53.824935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c63e27-a8e0-4e48-81ef-9b228ed48dad None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:53.858006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:53.858541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:53.858819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:53.859089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:53.859736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:53.860036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:53.861389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:53.869051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 582/1142] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:53.874657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:53.952421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:53.971682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7a06-3c83-4453-98df-9e0b8a6fbfda None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:54.004701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:54.005306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:54.005581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:54.005847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] PATH_INFO: `/v3/users/90e4947495594ed6ad92af3cf63d8a7c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:54.018224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] RBAC: Authorizing `identity:get_user(user_id=90e4947495594ed6ad92af3cf63d8a7c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:54.019646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:54.021777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 561/1143] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:53 2022] GET /identity/v3/users/90e4947495594ed6ad92af3cf63d8a7c => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:54.027533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:54.049724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:54.125049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27cf58-f28b-4021-8cc9-78ed51f29fef None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:54.157991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:54.158597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:54.158875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:54.159148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:54.223330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:54.224776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:54.244308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:54.320359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 583/1144] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:49:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:49:54.326229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:54.422204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def9abc-1a84-45aa-a608-794f7d0fa284 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:54.455139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:54.455755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:54.456034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:54.456329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] PATH_INFO: `/v3/projects/593b513cc7cb42028c37abf3cee076bc` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:54.457435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] RBAC: Authorizing `identity:get_project(project_id=593b513cc7cb42028c37abf3cee076bc)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:54.458905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b9ad3e-3fb5-40e5-8be1-12e601e34b2e None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:54.460495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 562/1145] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:54 2022] GET /identity/v3/projects/593b513cc7cb42028c37abf3cee076bc => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:54.466383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:54.563731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc9d19-d73b-4195-a264-75a1254bf01d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:54.583708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:54.597476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:54.598115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:54.598415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:54.598692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:54.599414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=593b513cc7cb42028c37abf3cee076bc, user.id=90e4947495594ed6ad92af3cf63d8a7c) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:54.599726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:54.601152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b865c34-744b-4994-bce7-58c31bd10ca5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:54.606719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 584/1146] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:49:54 2022] GET /identity/v3/role_assignments?user.id=90e4947495594ed6ad92af3cf63d8a7c&scope.project.id=593b513cc7cb42028c37abf3cee076bc&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:56.703139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945725c-cbb3-49df-b76a-a01246f03a2c None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:56.703679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945725c-cbb3-49df-b76a-a01246f03a2c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:56.703679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e945725c-cbb3-49df-b76a-a01246f03a2c None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:56.705020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 563/1147] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:56.710384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:56.710814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:56.710909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:56.728587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:56.729761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:56.793875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 585/1148] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:49:57.055545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e945725c-cbb3-49df-b76a-a01246f03a2c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:57.064195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:57.154553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e945725c-cbb3-49df-b76a-a01246f03a2c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:57.187534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:57.188150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:57.188428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:57.188699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:57.198496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:49:57.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:49:57.204365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:49:57.205366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 564/1149] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:49:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:49:57.210236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:57.305746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522a478a-8e60-4299-a89a-11b73b56373f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:57.319129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:57.338728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:57.339333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:57.339611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:57.339883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:57.340538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:49:57.340839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:57.342226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:57.350516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 586/1150] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:49:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:57.356667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:57.422031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:57.455230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed263478-8e6d-4a7c-bd7a-b4149830f654 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:57.489365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:57.490016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:57.490290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:57.490562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] PATH_INFO: `/v3/users/90e4947495594ed6ad92af3cf63d8a7c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:57.491665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] RBAC: Authorizing `identity:get_user(user_id=90e4947495594ed6ad92af3cf63d8a7c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:57.493091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:57.494668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 565/1151] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:49:57 2022] GET /identity/v3/users/90e4947495594ed6ad92af3cf63d8a7c => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:49:57.500325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:57.595551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19e978-79f5-4c60-ab2e-15b266cab6e4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:57.627792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:57.628407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:57.628684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:57.628951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] PATH_INFO: `/v3/projects/593b513cc7cb42028c37abf3cee076bc` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:57.630126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] RBAC: Authorizing `identity:get_project(project_id=593b513cc7cb42028c37abf3cee076bc)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:57.631542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4bf8d2-1d47-48f6-8d02-633e193668e8 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:57.633134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 587/1152] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:49:57 2022] GET /identity/v3/projects/593b513cc7cb42028c37abf3cee076bc => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:49:57.638838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:49:57.734799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa74a2-a4dd-468a-8f42-6bd1c2483863 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:49:57.766856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:49:57.767490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:57.767765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:57.768030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] PATH_INFO: `/v3/projects/593b513cc7cb42028c37abf3cee076bc/users/90e4947495594ed6ad92af3cf63d8a7c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:57.770060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] RBAC: Authorizing `identity:create_grant(project_id=593b513cc7cb42028c37abf3cee076bc, user_id=90e4947495594ed6ad92af3cf63d8a7c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:49:57.772053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39679b05-89db-4f79-905a-59f618411d99 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:49:57.774175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:49:57.785533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 566/1153] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:49:57 2022] PUT /identity/v3/projects/593b513cc7cb42028c37abf3cee076bc/users/90e4947495594ed6ad92af3cf63d8a7c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:49:59.869581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cccbde-db49-4085-9da9-e5246711db45 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:59.870135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cccbde-db49-4085-9da9-e5246711db45 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:59.870226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cccbde-db49-4085-9da9-e5246711db45 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:59.871497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 588/1154] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:49:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:49:59.876806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:49:59.877223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:49:59.877358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:49:59.895168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:49:59.896856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:49:59.963459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 567/1155] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:49:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:00.232953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cccbde-db49-4085-9da9-e5246711db45 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:00.278750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:00.332750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cccbde-db49-4085-9da9-e5246711db45 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:00.366322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:00.366958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:00.367241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:00.367516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:00.377954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:50:00.382793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:50:00.383878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:50:00.384903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 589/1156] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:50:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:50:00.389872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:00.484759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df79a5-5f4b-4e39-878f-cd6af87dfbe4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:00.507774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:00.518978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:00.519577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:00.519856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:00.520128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:00.520811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:00.521112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:00.522512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:00.530251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 568/1157] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:50:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:00.535978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:00.632408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda5d66-725d-4311-a891-89b293b1ad94 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:00.634638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:00.664031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:00.664650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:00.664930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:00.665206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] PATH_INFO: `/v3/users/90e4947495594ed6ad92af3cf63d8a7c` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:00.666336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] RBAC: Authorizing `identity:get_user(user_id=90e4947495594ed6ad92af3cf63d8a7c)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:00.667757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:00.669307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 590/1158] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:50:00 2022] GET /identity/v3/users/90e4947495594ed6ad92af3cf63d8a7c => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:00.674946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:00.771423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970bc62d-b23e-4b01-b12f-16da1c156391 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:00.804050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:00.804640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:00.804920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:00.805191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:00.856835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:00.870289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:00.871716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:00.965268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 569/1159] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:50:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:50:00.971268 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:01.067909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66024b-271b-4327-b6ae-7602c1d4668c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:01.099747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:01.100364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:01.100642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:01.100913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] PATH_INFO: `/v3/projects/593b513cc7cb42028c37abf3cee076bc` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:01.102018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] RBAC: Authorizing `identity:get_project(project_id=593b513cc7cb42028c37abf3cee076bc)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:01.103426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6aa608-5e13-44fb-8991-7e25cc79a2c4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:01.105020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 591/1160] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:50:00 2022] GET /identity/v3/projects/593b513cc7cb42028c37abf3cee076bc => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:50:01.111363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:01.194059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:01.209196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acec186a-18af-4bc0-8c8d-fc04e0f1f628 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:01.242447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:01.243045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:01.243329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:01.243594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:01.244314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=593b513cc7cb42028c37abf3cee076bc, user.id=90e4947495594ed6ad92af3cf63d8a7c) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:01.244617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:01.246099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8010d5-9942-4d2e-87b8-a5ea340625ba None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:01.251875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 570/1161] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:50:01 2022] GET /identity/v3/role_assignments?user.id=90e4947495594ed6ad92af3cf63d8a7c&scope.project.id=593b513cc7cb42028c37abf3cee076bc&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:03.402966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db973deb-fb4e-4543-91e7-2d5da29312c7 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:03.403549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db973deb-fb4e-4543-91e7-2d5da29312c7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:03.403549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db973deb-fb4e-4543-91e7-2d5da29312c7 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:03.404830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 592/1162] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:03.410047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:03.410445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:03.410599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:03.428768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:03.429950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:03.494643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 571/1163] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:03.762452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db973deb-fb4e-4543-91e7-2d5da29312c7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:03.805186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:03.861133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db973deb-fb4e-4543-91e7-2d5da29312c7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:03.892766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:03.893352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:03.893625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:03.893892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:03.956884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:03.958390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:03.996658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:04.052483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 593/1164] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:50:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:50:04.058489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:04.135738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:04.154747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175d8d2-bb1a-4de1-923b-b2f9fa6c8235 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:04.187676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:04.188303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:04.188584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:04.188857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:04.195443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:04.196875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-80aa3176-cf2b-425b-8df3-a326ad340b87 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:50:04.208609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:50:04.209852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:50:04.211052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 572/1165] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:50:04 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:50:04.215652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:04.312475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4c4880-5781-4912-8f8c-a5e54f9aa14d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:04.345086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:04.345697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:04.346025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:04.346300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:04.346935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:04.347237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:04.349196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ea370-2298-4428-bac9-c4b0116601da None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:04.356371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 594/1166] 10.222.0.46 () {56 vars in 1115 bytes} [Fri Oct 28 10:50:04 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:06.446725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4919bd8a-05f5-4acd-a50c-4c84083c0171 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:06.447272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4919bd8a-05f5-4acd-a50c-4c84083c0171 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:06.447272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4919bd8a-05f5-4acd-a50c-4c84083c0171 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:06.448595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 573/1167] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:06.453764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cd279-2832-4c42-9a24-494bb79044b2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:06.454213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cd279-2832-4c42-9a24-494bb79044b2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:06.454334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653cd279-2832-4c42-9a24-494bb79044b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:06.461490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:06.472550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-653cd279-2832-4c42-9a24-494bb79044b2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:06.473749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-653cd279-2832-4c42-9a24-494bb79044b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:06.537087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 595/1168] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:06.812789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4919bd8a-05f5-4acd-a50c-4c84083c0171 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:06.912408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4919bd8a-05f5-4acd-a50c-4c84083c0171 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:06.944661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:06.945259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:06.945540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:06.945812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:06.946510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:06.947866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862b40d-88f0-4acc-8c85-df18616c20ed None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:06.958149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:06.965148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 574/1169] 10.222.0.46 () {60 vars in 1138 bytes} [Fri Oct 28 10:50:06 2022] POST /identity/v3/domains => generated 267 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:50:09.095715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d023aa90-abf6-4975-acd4-6880f5d863cd None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:09.096247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d023aa90-abf6-4975-acd4-6880f5d863cd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:09.096247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d023aa90-abf6-4975-acd4-6880f5d863cd None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:09.097606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 596/1170] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:09.103240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:09.103667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:09.103775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:09.122389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:09.123566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:09.187054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 575/1171] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:09.455518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d023aa90-abf6-4975-acd4-6880f5d863cd None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:09.552841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d023aa90-abf6-4975-acd4-6880f5d863cd None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:09.585120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:09.585735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:09.586039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:09.586315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] PATH_INFO: `/v3/domains/5578ddff8c774086a47d36b1067892d6` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:09.588164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:09.596742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=5578ddff8c774086a47d36b1067892d6)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:09.598799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8128a4e0-66b2-482e-bd52-21d77443db53 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:09.600394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 597/1172] 10.222.0.46 () {56 vars in 1183 bytes} [Fri Oct 28 10:50:09 2022] GET /identity/v3/domains/5578ddff8c774086a47d36b1067892d6 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:09.606394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:09.702619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c7e5f0-6995-4604-810b-cf74890f1327 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:09.731676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:09.737655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:09.738282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:09.738563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:09.738838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:09.739478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:09.740903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d698cd9b-5694-4666-87b1-7b57da957f2c None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:09.765126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 576/1173] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:50:09 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:50:11.914157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8121863-b396-480b-b27f-ba0be6e1b8b4 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:11.914716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8121863-b396-480b-b27f-ba0be6e1b8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:11.914716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8121863-b396-480b-b27f-ba0be6e1b8b4 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:11.916072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 598/1174] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:11.921186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:11.921619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:11.921729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:11.939382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:11.940542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:12.004015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 577/1175] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:12.270661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8121863-b396-480b-b27f-ba0be6e1b8b4 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:12.369338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8121863-b396-480b-b27f-ba0be6e1b8b4 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:12.402046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:12.402662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:12.402939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:12.403211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] PATH_INFO: `/v3/domains/5578ddff8c774086a47d36b1067892d6` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:12.404308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] RBAC: Authorizing `identity:get_domain(domain_id=5578ddff8c774086a47d36b1067892d6)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:12.405721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34aa5d1b-357a-4c61-b733-eda9c1a7c16f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:12.407719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 599/1176] 10.222.0.46 () {56 vars in 1183 bytes} [Fri Oct 28 10:50:12 2022] GET /identity/v3/domains/5578ddff8c774086a47d36b1067892d6 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:12.413487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:12.508146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464b852-6b88-405f-b6ec-3565c5adc3fa None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:12.520618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:12.541476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:12.542036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:12.542312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:12.542581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:12.543225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:12.544642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7be6ca7-a9e1-413a-aef4-db005d6e55ba None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:12.564928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 578/1177] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:50:12 2022] POST /identity/v3/users => generated 311 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:50:14.721422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb834735-06c7-4d45-9461-6b10e5c994ba None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:14.721953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb834735-06c7-4d45-9461-6b10e5c994ba None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:14.721953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb834735-06c7-4d45-9461-6b10e5c994ba None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:14.723935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 600/1178] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:14.729243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:14.729683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:14.729784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:14.747838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:14.749017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:14.812762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 579/1179] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:15.077788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb834735-06c7-4d45-9461-6b10e5c994ba None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:15.137820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:15.175939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb834735-06c7-4d45-9461-6b10e5c994ba None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:15.208387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:15.209006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:15.209285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:15.209553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:15.219667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:50:15.224473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:50:15.225572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:50:15.226837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 601/1180] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:50:15.231605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:15.323046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:15.327226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70793a04-3af9-44d3-8db9-d75cc9b12d29 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:15.361321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:15.361863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:15.362173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:15.362445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:15.363102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:15.363404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:15.364891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:15.372624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 580/1181] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:15.378523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:15.475673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de5734-4963-400e-9097-dfe2e79c37ab None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:15.497308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:15.509311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:15.509928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:15.510238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:15.510510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] PATH_INFO: `/v3/users/39a3668dc08f490a80784650ae12a99c` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:15.523027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] RBAC: Authorizing `identity:get_user(user_id=39a3668dc08f490a80784650ae12a99c)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:15.524452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:15.526037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 602/1182] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/users/39a3668dc08f490a80784650ae12a99c => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:50:15.531692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:15.628524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05207b8e-ac51-4820-815a-f186669a3ee9 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:15.660678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:15.661269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:15.661547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:15.661819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:15.690161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:15.728750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:15.730227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:15.825472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 581/1183] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:50:15.831350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:15.927331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e98110b-b974-4e3f-8a1d-a5227e1cd38a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:15.960392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:15.961040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:15.961321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:15.961596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] PATH_INFO: `/v3/projects/588a2561fcc541d79c379e584c230721` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:15.962728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] RBAC: Authorizing `identity:get_project(project_id=588a2561fcc541d79c379e584c230721)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:15.964150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f32b35-7e58-49c4-8999-12cfa2d55ed0 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:15.965895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 603/1184] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/projects/588a2561fcc541d79c379e584c230721 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:15.971577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:16.027725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:16.065752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d58ccf-60e7-493b-baa4-c08d5342b285 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:16.099163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:16.099760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:16.100041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:16.100314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:16.101030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=588a2561fcc541d79c379e584c230721, user.id=39a3668dc08f490a80784650ae12a99c) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:16.101339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:16.102792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3083ed-10da-4995-aa82-db1cf193d018 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:16.108291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 582/1185] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:50:15 2022] GET /identity/v3/role_assignments?user.id=39a3668dc08f490a80784650ae12a99c&scope.project.id=588a2561fcc541d79c379e584c230721&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:18.178121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83beffa0-e435-45b2-bd91-87b515d3db5e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:18.178676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83beffa0-e435-45b2-bd91-87b515d3db5e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:18.178676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83beffa0-e435-45b2-bd91-87b515d3db5e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:18.179990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 604/1186] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:18.184972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:18.185386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:18.185519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:18.203815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:18.204998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:18.270452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 583/1187] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:18.543681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83beffa0-e435-45b2-bd91-87b515d3db5e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:18.592272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:18.642619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83beffa0-e435-45b2-bd91-87b515d3db5e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:18.674384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:18.675001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:18.675276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:18.675542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:18.685719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:50:18.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:50:18.691525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:50:18.692962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 605/1188] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:50:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:50:18.698030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:18.750611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:18.793810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9fc109-d284-4389-8eee-4a7be2798988 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:18.825686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:18.826307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:18.826583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:18.826849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:18.827502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:18.827806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:18.829154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:18.837230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 584/1189] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:50:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:18.842832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:18.938633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7956d57-559d-446d-b63e-db3f0d73c383 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:18.947334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:18.970628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:18.971238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:18.971512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:18.971782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] PATH_INFO: `/v3/users/39a3668dc08f490a80784650ae12a99c` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:18.973387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] RBAC: Authorizing `identity:get_user(user_id=39a3668dc08f490a80784650ae12a99c)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:18.974812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:18.976333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 606/1190] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:50:18 2022] GET /identity/v3/users/39a3668dc08f490a80784650ae12a99c => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:18.982314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:19.077279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6494a65-11fe-454c-ae85-d8e3998f9333 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:19.092648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:19.110565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:19.111172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:19.111448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:19.111718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] PATH_INFO: `/v3/projects/588a2561fcc541d79c379e584c230721` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:19.112813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] RBAC: Authorizing `identity:get_project(project_id=588a2561fcc541d79c379e584c230721)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:19.114828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d1fef-b82a-4c26-bc1c-b76df6219380 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:19.116388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 585/1191] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:50:18 2022] GET /identity/v3/projects/588a2561fcc541d79c379e584c230721 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:19.122140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:19.217240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1b8e8d-b3a6-41ba-ad55-bcc0f9944a13 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:19.249785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:19.250454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:19.250730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:19.250999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] PATH_INFO: `/v3/projects/588a2561fcc541d79c379e584c230721/users/39a3668dc08f490a80784650ae12a99c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:19.252843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] RBAC: Authorizing `identity:create_grant(project_id=588a2561fcc541d79c379e584c230721, user_id=39a3668dc08f490a80784650ae12a99c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:19.254283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addb24bf-feab-43b1-82f2-09e294631eb9 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:19.266439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 607/1192] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:50:19 2022] PUT /identity/v3/projects/588a2561fcc541d79c379e584c230721/users/39a3668dc08f490a80784650ae12a99c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:50:21.367846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869d586e-6355-45d5-9d08-bc622e4bab1a None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:21.368366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869d586e-6355-45d5-9d08-bc622e4bab1a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:21.368456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869d586e-6355-45d5-9d08-bc622e4bab1a None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:21.369729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 586/1193] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:50:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:50:21.375028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:21.375463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:21.375579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:21.393123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:50:21.394335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:50:21.422586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:21.460832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 608/1194] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:50:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:50:21.733952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869d586e-6355-45d5-9d08-bc622e4bab1a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:21.833282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869d586e-6355-45d5-9d08-bc622e4bab1a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:21.866372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:21.866980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:21.867257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:21.867524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:21.877686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:50:21.882579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:50:21.883661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:50:21.884502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 587/1195] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:50:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:50:21.889614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:21.985100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51c3f6c-9b6a-4c47-a27e-fae33efdc6f7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:22.017444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:22.018065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:22.018346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:22.018618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:22.019269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:22.019570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:22.020952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:22.028597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 609/1196] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:50:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:22.034344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:22.078781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:22.131910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc3ea9f-bce7-4fde-90b8-e23f8f57992b None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:22.164539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:22.165154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:22.165430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:22.165699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] PATH_INFO: `/v3/users/39a3668dc08f490a80784650ae12a99c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:22.166832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] RBAC: Authorizing `identity:get_user(user_id=39a3668dc08f490a80784650ae12a99c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:22.168245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:22.169768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 588/1197] 10.222.0.46 () {56 vars in 1177 bytes} [Fri Oct 28 10:50:22 2022] GET /identity/v3/users/39a3668dc08f490a80784650ae12a99c => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:22.175632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:22.202942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:22.272622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a134d9a-23f0-4826-9ed5-f2d58db694c5 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:22.304763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:22.305363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:22.305641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:22.305912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:22.369768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:22.371226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:22.396579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:22.465919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 610/1198] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:50:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:50:22.471711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:22.568213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4167f0-3582-4e32-a713-544e78f45742 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:22.580245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:22.601180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:22.601795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:22.602098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:22.602369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] PATH_INFO: `/v3/projects/588a2561fcc541d79c379e584c230721` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:22.603471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] RBAC: Authorizing `identity:get_project(project_id=588a2561fcc541d79c379e584c230721)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:22.604881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909dcfc5-188c-4160-bdda-33c868b9b777 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:22.606817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 589/1199] 10.222.0.46 () {56 vars in 1186 bytes} [Fri Oct 28 10:50:22 2022] GET /identity/v3/projects/588a2561fcc541d79c379e584c230721 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:22.612740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:50:22.709772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34f810-1a7f-48fe-a7f9-2da0ff1a50fd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:50:22.735566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:22.744332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:50:22.744919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:50:22.745193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:50:22.745460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:50:22.746212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=588a2561fcc541d79c379e584c230721, user.id=39a3668dc08f490a80784650ae12a99c) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:50:22.746513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:50:22.747927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e2f6e7-bdbf-420b-a974-377c2fe72d25 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:50:22.753422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 611/1200] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:50:22 2022] GET /identity/v3/role_assignments?user.id=39a3668dc08f490a80784650ae12a99c&scope.project.id=588a2561fcc541d79c379e584c230721&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:50:32.597120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=10 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:50:32.750984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:17.992004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6564a-0d63-4ff1-99ea-00d55b83d530 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:17.992631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6564a-0d63-4ff1-99ea-00d55b83d530 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:17.992631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6564a-0d63-4ff1-99ea-00d55b83d530 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:17.994320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 590/1201] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:18.000480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:18.001042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:18.001042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:18.023959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:18.025197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:18.110284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 612/1202] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:18.394400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6564a-0d63-4ff1-99ea-00d55b83d530 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:18.517495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6564a-0d63-4ff1-99ea-00d55b83d530 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:18.556546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:18.557111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:18.557382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:18.557656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:18.569992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:18.571495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:18.587839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:18.588993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:18.590022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:18.590022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:18.590022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:18.590022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 591/1203] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:51:18.597902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:18.728450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a7b977-4d34-44c6-949b-5bb867eba586 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:18.767961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:18.782989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:18.783764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:18.784708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:18.784987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:18.785803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:18.786117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:18.787739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bddede5-5ecf-490f-9f5b-529bd9c95835 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:18.801316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 613/1204] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:18.808311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:18.883054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:18.945901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747a06b1-9e53-4aaf-a6f1-172dca5503dd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:18.987447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:18.988426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:18.988724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:18.989065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:18.989664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:18.991267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da190d1b-8ebb-4273-a158-239be640c3f1 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:19.023590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 592/1205] 10.222.0.46 () {60 vars in 1132 bytes} [Fri Oct 28 10:51:18 2022] POST /identity/v3/users => generated 251 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:51:21.530248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75600566-fd75-474c-ab11-51a47c2366ee None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:21.530805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75600566-fd75-474c-ab11-51a47c2366ee None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:21.530805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75600566-fd75-474c-ab11-51a47c2366ee None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:21.534487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 614/1206] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:21.538451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:21.538872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:21.538980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:21.558095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:21.559296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:21.640366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 593/1207] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:21.912251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75600566-fd75-474c-ab11-51a47c2366ee None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.036265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75600566-fd75-474c-ab11-51a47c2366ee None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.075601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.076294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.076573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.076845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:22.089697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:22.095385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:22.096599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:22.097536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 615/1208] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:51:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:22.102670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.151444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:22.221545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd575a2-c3f2-40bd-ba31-3fed57da30b3 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.269807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.270561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.270931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.271666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:22.271977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:22.272290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:22.273693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:22.283228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 594/1209] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:51:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:51:22.289286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.350670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:22.415026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae99a4fe-48d1-46a1-b7b3-fe75eed524ae None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.454792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.457153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.457153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.457153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:22.471513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:22.473653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:22.490550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:22.491764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:22.492878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:22.492878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:22.492878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:22.492878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 616/1210] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:51:22.497931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.604506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:22.608983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b514ff7-b1a5-4c27-aee0-e27436148800 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.646278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.646910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.647195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.647462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:22.648170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:22.648479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:22.649936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:22.667675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 595/1211] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:22.669544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.778364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef7d1a-8993-4fcb-a37a-603a9b7a4f89 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.779972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:22.816754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.817420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.817709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.818031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:22.828002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:22.829518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:22.841796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:22.843024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:22.844174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:22.846106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 617/1212] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:22.849541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:22.963066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae31bba8-6ac9-4983-9718-079ace2649f6 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:22.998333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:22.999071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:22.999447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:22.999799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:23.000728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:23.001053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:23.002596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:23.020401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 596/1213] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:22 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:23.026201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:23.145178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b5fde3-892f-4ef5-b3e9-f79f79039f01 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:23.183880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:23.184488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:23.187277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:23.187277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:23.225458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:23.278542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:23.280112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:23.397395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 618/1214] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:51:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:51:23.403313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:23.423854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:23.515860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47583a31-6023-4d3f-82ab-ffb93e049144 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:23.554451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:23.555148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:23.555424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:23.555690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:23.570114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:23.571985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:23.591680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:23.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:23.593682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:23.593682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:23.593682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:23.593682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 597/1215] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:51:23.599701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:23.678882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:23.722303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83382474-4d3d-44df-a014-d4b89c2ac3ed None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:23.760907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:23.761514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:23.761827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:23.762113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:23.762764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:23.763067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:23.764431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:23.776507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 619/1216] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:23.783123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:23.864235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:23.897632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5819d91d-5acb-4ed6-b19e-5bac943e8b05 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:23.938245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:23.938670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:23.938985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:23.939247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:23.947969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:23.949943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:23.963680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:23.965145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:23.966516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:23.972256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 598/1217] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:23.972381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:24.086609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d7ee8d-1d97-4c4c-8439-7201722a2bfd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:24.109343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:24.122074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:24.122684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:24.122980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:24.123231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:24.123910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:24.124209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:24.125796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b063628f-8a5d-42bb-a640-68e92b519687 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:24.140168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 620/1218] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:24.147843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:24.259167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd95387c-098d-4aea-86d4-620ede3d4895 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:24.295159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:24.301899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:24.302667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:24.302985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:24.303260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:24.304034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=73aefca6b8d04a0e8bd2fad4ffedbccd) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:24.304300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:24.305762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9932dca-2969-404d-8225-af395a082c46 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:24.311578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 599/1219] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:51:24 2022] GET /identity/v3/role_assignments?user.id=73aefca6b8d04a0e8bd2fad4ffedbccd&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:26.459594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446cae88-2a5e-42da-b74c-c5a3e06c5789 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:26.460156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446cae88-2a5e-42da-b74c-c5a3e06c5789 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:26.460156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446cae88-2a5e-42da-b74c-c5a3e06c5789 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:26.461499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 621/1220] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:26.466880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:26.467305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:26.467402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:26.486256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:26.487756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:26.558572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 600/1221] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:26.849778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446cae88-2a5e-42da-b74c-c5a3e06c5789 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:26.973843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446cae88-2a5e-42da-b74c-c5a3e06c5789 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.009806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.010468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.010752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.011033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.022221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:27.027023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:27.028335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:27.029553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:27.033061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 622/1222] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:51:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:27.034992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:27.136865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ca3fc4-3aa9-4eae-a765-59b577095466 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.160704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:27.171777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.172381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.172653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.172969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.173590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] RBAC: Adding query filter params (name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:27.173889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:27.175305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:27.183227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 601/1223] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:27.189424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:27.236137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:27.308729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72649a74-ff19-46e5-8fbe-ac2670cc080c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.345719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.346409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.346669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.346944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.357408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:27.358894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:27.373550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:27.374746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:27.376031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:27.376031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:27.376031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:27.376031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 623/1224] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:27.381139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:27.495331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b06503-34d8-4e59-9c9c-1d4b33189a2d None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.530943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.531551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.531832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.532117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.532746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:27.533046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:27.534565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:27.548388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 602/1225] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:27.555839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:27.656824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:27.664665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d19793-8a0c-48df-adf0-a5744354b138 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.702816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.703446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.703721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.703994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] PATH_INFO: `/v3/users/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.711457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:27.712854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:27.724305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:27.725688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:27.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 624/1226] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:51:27.731665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:27.784292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:27.854887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593d78c-bc26-4912-b50b-0fbbfbbeaddd None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:27.897585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:27.898233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:27.898510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:27.898779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:27.899500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:27.899848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:27.901391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:27.918724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 603/1227] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:27.924692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:28.046887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcbafe-2d5a-48b2-85cb-f887b5d7f5de None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:28.090604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:28.091237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:28.091523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:28.091779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:28.104932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:28.106410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:28.122253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:28.123412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:28.124463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:28.124463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:28.124463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:28.124463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 625/1228] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:28.130710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:28.238312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae24bac-ca7a-4369-a5e9-6b5ca8de1459 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:28.241035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:28.280364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:28.280982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:28.281267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:28.281520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:28.282222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:28.282486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:28.284485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:28.295883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 604/1229] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:28.301782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:28.313886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:28.405075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2b8ad5-4361-4472-b0d1-4c50d654f6d2 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:28.439844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:28.440546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:28.440818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:28.441085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:28.448109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:28.449549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:28.461296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:28.462423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:28.463577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 626/1230] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:28.468391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:28.571895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed11c4-2afe-45cd-89c5-cddf6505c722 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:28.606069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:28.606680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:28.607227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:28.607501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:28.608191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:28.608494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:28.609943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e2eb2-6964-44e1-8a77-48c89aeba9c4 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:28.622642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 605/1231] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:28.629312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:28.699290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:28.735641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3f4990-610c-46a8-8d1f-69df1bd4f184 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:28.772403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:28.773085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:28.773349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:28.773620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/73aefca6b8d04a0e8bd2fad4ffedbccd/roles/09c58019bdac4dfab204b31d1563523b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:28.788301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=73aefca6b8d04a0e8bd2fad4ffedbccd, role_id=09c58019bdac4dfab204b31d1563523b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:28.789772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4286300-d453-4467-9f14-7f880e43b05c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:28.802671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 627/1232] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:51:28 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/73aefca6b8d04a0e8bd2fad4ffedbccd/roles/09c58019bdac4dfab204b31d1563523b => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:51:31.159716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5bd98d-4919-412d-963f-9f3f41c611a2 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:31.160275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5bd98d-4919-412d-963f-9f3f41c611a2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:31.160275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5bd98d-4919-412d-963f-9f3f41c611a2 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:31.161572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 606/1233] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:51:31.168266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:31.168711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:31.168811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:31.189060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:31.190828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:31.257721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 628/1234] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:31.566636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5bd98d-4919-412d-963f-9f3f41c611a2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:31.591783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:31.672411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5bd98d-4919-412d-963f-9f3f41c611a2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:31.709482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:31.710161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:31.710439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:31.710707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:31.721750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:31.726564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:31.727688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:31.728562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 607/1235] 10.222.0.46 () {56 vars in 1102 bytes} [Fri Oct 28 10:51:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:31.734606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:31.785016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:31.844623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea9188c-1b85-489b-80a3-2b30da147726 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:31.884289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:31.885032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:31.885357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:31.885679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:31.886463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] RBAC: Adding query filter params (name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:31.886766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:31.888375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:31.898031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 629/1236] 10.222.0.46 () {56 vars in 1103 bytes} [Fri Oct 28 10:51:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:31.904377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:32.005226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:32.021071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d93344b-10d7-43db-a6f3-f927693882d2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:32.057731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:32.058441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:32.058742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:32.059113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:32.070061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:32.071512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:32.092210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:32.093429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:32.094431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:32.094431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:32.094431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:32.094431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 608/1237] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:32.101034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:32.210373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4766e9c-f1d8-4880-aa69-1608d1208a5c None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:32.220257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:32.251329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:32.251990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:32.252271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:32.252642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:32.253299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:32.253645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:32.255046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:32.271226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 630/1238] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:32.276998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:32.387645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f447b17-3732-4c94-a002-411cfa09c7f0 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:32.423663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:32.424427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:32.424715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:32.424989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:32.428315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:32.434400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:32.436153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:32.448975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:32.450215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:32.454471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 609/1239] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:32.456802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:32.567931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a1f2-a41e-457a-bd52-a0f20ab05fc4 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:32.605646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:32.606348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:32.606625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:32.607176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:32.607920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:32.608271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:32.609785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:32.628181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 631/1240] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:32.634659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:32.745279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630fe1ce-65c5-4b03-b012-ad8dbb29bd15 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:32.782442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:32.783063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:32.783499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:32.783885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:32.854615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:32.865501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:32.867127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:32.995026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 610/1241] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:51:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:51:32.996360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:33.025371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:33.103507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26705ce1-0683-4bd3-b133-87dabdfdb9aa None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:33.141838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:33.142503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:33.142795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:33.143077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:33.153572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:33.155050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:33.170217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:33.171221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:33.172111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:33.172111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:33.172111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:33.176976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 632/1242] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:33.177879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:33.278238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:33.285612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781dd47f-92a2-46e8-8b4d-fa10d2f4da1a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:33.320393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:33.321001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:33.321278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:33.321555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:33.322225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:33.322530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:33.323911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:33.335270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 611/1243] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:51:33.340943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:33.426158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:33.448533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f21d0ec-bba3-4909-a1b4-d13934bf3e11 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:33.484716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:33.485375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:33.485675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:33.485947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:33.492941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:33.494502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:33.506083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:33.507010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:33.507915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 633/1244] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:33.513128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:33.623611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf32629-d63c-4db8-acf7-cda65d53d999 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:33.660181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:33.660825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:33.661121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:33.661446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:33.662239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:33.662545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:33.664085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d430312-ad75-4680-a1c4-d312ade4b9bd None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:33.677358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 612/1245] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:51:33.683423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:33.793990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9e969-1012-449a-8779-4c6d37199485 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:33.832005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:33.832774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:33.833110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:33.833454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:33.834566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] RBAC: Adding query filter params (role.id=09c58019bdac4dfab204b31d1563523b, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=73aefca6b8d04a0e8bd2fad4ffedbccd) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:33.835066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:33.837223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332bccc7-efe4-466d-ae9b-30960f33ce82 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:33.846742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 634/1246] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:51:33 2022] GET /identity/v3/role_assignments?user.id=73aefca6b8d04a0e8bd2fad4ffedbccd&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=09c58019bdac4dfab204b31d1563523b => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:36.138606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9413c2a4-db11-429a-bf68-f43c202e5103 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:36.139170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9413c2a4-db11-429a-bf68-f43c202e5103 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:36.139266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9413c2a4-db11-429a-bf68-f43c202e5103 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:36.140749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 613/1247] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:36.146725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:36.147194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:36.147306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:36.153477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:36.166844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:36.168148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:36.241501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 635/1248] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:36.527122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9413c2a4-db11-429a-bf68-f43c202e5103 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:36.546388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:36.635256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9413c2a4-db11-429a-bf68-f43c202e5103 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:36.672856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:36.673502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:36.673803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:36.674102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:36.685332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:36.690185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:36.691289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:36.692301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 614/1249] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:51:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 10:51:36.697619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:36.802897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b7123c-da9b-45c4-9cda-38f34e9c573f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:36.836166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:36.843202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:36.843901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:36.844187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:36.844493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:36.845241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:36.845528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:36.847098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:36.855967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 636/1250] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:51:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:36.862782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:36.948749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:36.975632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676002b1-a53e-4896-aec3-054a35b0aae2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.015242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:37.015885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:37.016167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:37.016438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:37.026749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:37.028209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:37.043946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:37.044916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:37.046087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:37.046087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:37.046087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:37.046087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 615/1251] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:37.051440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:37.170061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f2519-28ee-4bf6-9092-7a9b58729c17 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.204741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:37.205885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:37.206176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:37.206445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:37.207095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:37.207394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:37.208778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:37.222473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 637/1252] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:37.228376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:37.335133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb4ff0-ced6-4298-b8f5-27d7fcafde4f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.362891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:37.371691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:37.372348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:37.372629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:37.372905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:37.381460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:37.382978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:37.395717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:37.396745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:37.397670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 616/1253] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:37.403023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:37.461132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:37.508552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62befc1f-06c2-42e2-b32c-2171041f2968 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.543274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:37.543885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:37.544164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:37.544440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:37.545163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:37.545466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:37.546918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:37.564564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 638/1254] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:37.570397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:37.673730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349c5e1c-348c-4923-9c4c-75a8c82137ee None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.706834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:37.707437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:37.707711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:37.707978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:37.762685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:37.775896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:37.777360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:37.873455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 617/1255] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:51:37 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:51:37.879805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:37.978970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0d067-275b-4a1d-adc6-e7c298d7cc5d None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:37.992422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:38.013034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:38.013591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:38.013865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:38.014159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:38.023663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:38.025076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:38.039062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:38.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:38.041238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:38.041238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:38.041238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:38.042126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 639/1256] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:38.045823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:38.130542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:38.142575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25955e2a-3a30-4452-aefe-720f31ae4689 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:38.175675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:38.176266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:38.176546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:38.176818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:38.177450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:38.177750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:38.179151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:38.190581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 618/1257] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:38.196159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:38.292544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68482a6-e094-4e91-951d-f4cdd9c893ae None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:38.325451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:38.326097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:38.326376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:38.326647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:38.333057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:38.334517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:38.345353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:38.346934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:38.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 640/1258] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:38.352781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:38.449497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08ebdaa-36a0-421b-8614-11b680cd28e3 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:38.481876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:38.482756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:38.483038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:38.483313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:38.483998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:38.484301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:38.485720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a064effd-03a3-4028-b07e-59f5accc9a82 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:38.497942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 619/1259] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:38.503988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:38.528730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:38.600722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7020e8-2715-41f6-8eae-a10a6d428bc6 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:38.632917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:38.633506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:38.633782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:38.634075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:38.634792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=73aefca6b8d04a0e8bd2fad4ffedbccd) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:38.635096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:38.636515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd8e72-2cb0-4c07-a123-8dda815e8429 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:38.641687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 641/1260] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:51:38 2022] GET /identity/v3/role_assignments?user.id=73aefca6b8d04a0e8bd2fad4ffedbccd&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:40.728601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcaa978-54e2-4239-b7d4-e021c8dfa4f7 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:40.729124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcaa978-54e2-4239-b7d4-e021c8dfa4f7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:40.729124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcaa978-54e2-4239-b7d4-e021c8dfa4f7 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:40.730490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 620/1261] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:40.735733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:40.736155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:40.736262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:40.754626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:40.755788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:40.818740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 642/1262] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 10:51:41.087203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcaa978-54e2-4239-b7d4-e021c8dfa4f7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.098540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:41.186130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcaa978-54e2-4239-b7d4-e021c8dfa4f7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.218397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.219004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.219282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.219549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.229688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:41.234681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:41.235790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:41.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 621/1263] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:41.241646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.259433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:41.338289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ae604e-3e69-43c8-801e-bb79f5638dbd None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.370150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.370745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.371023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.371421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.372082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:41.372382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:41.373732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:41.382159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 643/1264] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:51:41.387773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.452248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:41.485494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9814b6e-8641-4f73-8d8c-043735beb38f None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.518510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.519123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.519399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.519666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.529707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:41.531181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:41.544825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:41.545922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:41.546974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:41.546974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:41.546974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:41.546974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 622/1265] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:41.551543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.630012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:41.649337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932e1387-1cce-4f42-9b1d-a2ef5af7d6c7 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.681284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.681885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.682189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.682459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.683099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:41.683397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:41.684736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:41.696613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 644/1266] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:41.702301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.797004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84224175-e8eb-4a86-a069-2750014dc574 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.816533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:41.830598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.831205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.831477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.831744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.839089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:41.840513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:41.851355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:41.852445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:41.853422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 623/1267] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:41.858430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:41.960603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5b9662-670e-4c02-b249-14893b502681 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:41.993563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:41.994199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:41.994477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:41.994747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:41.995465 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:41.995771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:41.997185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.006141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:42.014379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 645/1268] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:41 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 10:51:42.020164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:42.116407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c277533-755d-429e-8671-839528e7e0d2 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:42.147829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:42.148432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:42.148709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:42.148980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:42.159486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:42.160969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:42.174993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:42.176157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:42.177196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:42.177196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:42.177196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:42.177196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 624/1269] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:42.181580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:42.278180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2b7095-3b2d-4f55-82a8-ab842433281e None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:42.311295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613561f8-9897-4531-988f-147a652f462b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:42.311898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613561f8-9897-4531-988f-147a652f462b None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:42.312177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613561f8-9897-4531-988f-147a652f462b None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:42.312445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613561f8-9897-4531-988f-147a652f462b None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:42.313072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613561f8-9897-4531-988f-147a652f462b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:42.313374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613561f8-9897-4531-988f-147a652f462b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:42.314785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613561f8-9897-4531-988f-147a652f462b None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.325568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 646/1270] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:42.331121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:42.363450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:42.429053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794985e-bb4d-48e9-b179-c11e7dacd125 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:42.461001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:42.461611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:42.461887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:42.462180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:42.468561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:42.469982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:42.481401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:42.482702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:42.483996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 625/1271] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:42.489021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613561f8-9897-4531-988f-147a652f462b None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:42.544871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:42.585720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613561f8-9897-4531-988f-147a652f462b None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:42.617808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:42.618427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:42.618704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:42.618974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:42.619654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:42.619952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:42.621360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5c737c-11be-4ab6-a6a6-6d7098ea4ff6 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.633325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 647/1272] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:42.639319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:42.728765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:42.738388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582fdbb7-3ed0-49c9-8788-9532768f5286 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:42.770771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:42.771419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:42.771699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:42.771973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] PATH_INFO: `/v3/projects/f6834f875cac433ca5419f5b85701542/users/73aefca6b8d04a0e8bd2fad4ffedbccd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:42.773837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] RBAC: Authorizing `identity:create_grant(project_id=f6834f875cac433ca5419f5b85701542, user_id=73aefca6b8d04a0e8bd2fad4ffedbccd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:42.775297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17fc7f5-d6f3-4a42-907a-c4bcc969cda3 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:42.788360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 626/1273] 10.222.0.46 () {58 vars in 1439 bytes} [Fri Oct 28 10:51:42 2022] PUT /identity/v3/projects/f6834f875cac433ca5419f5b85701542/users/73aefca6b8d04a0e8bd2fad4ffedbccd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:51:44.908444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2b9f7-d81e-4cca-9018-fbf5032bbfbc None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:44.908995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2b9f7-d81e-4cca-9018-fbf5032bbfbc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:44.908995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2b9f7-d81e-4cca-9018-fbf5032bbfbc None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:44.910360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 648/1274] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:44.915522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:44.915923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:44.916073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:44.935061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:44.936245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:45.009682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 627/1275] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:45.274605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce2b9f7-d81e-4cca-9018-fbf5032bbfbc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:45.373660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce2b9f7-d81e-4cca-9018-fbf5032bbfbc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:45.391158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:45.407998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17223380-783f-4722-bc56-b66c09356419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:45.409167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17223380-783f-4722-bc56-b66c09356419 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:45.409445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17223380-783f-4722-bc56-b66c09356419 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:45.409717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17223380-783f-4722-bc56-b66c09356419 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:45.419566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17223380-783f-4722-bc56-b66c09356419 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-17223380-783f-4722-bc56-b66c09356419 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 10:51:45.424334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:45.425405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:45.427035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 649/1276] 10.222.0.46 () {56 vars in 1096 bytes} [Fri Oct 28 10:51:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:45.431437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:45.508243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:45.526968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0518a-836a-4a15-bb17-d9743fb8ff01 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:45.559089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:45.559677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:45.559952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:45.560222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:45.560874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:45.561173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:45.562554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:45.570461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 628/1277] 10.222.0.46 () {56 vars in 1099 bytes} [Fri Oct 28 10:51:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:45.576262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17223380-783f-4722-bc56-b66c09356419 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:45.672725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17223380-783f-4722-bc56-b66c09356419 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:45.704187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:45.704793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:45.705069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:45.705865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:45.715919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:45.717350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:45.731455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:45.732870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:45.733956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:45.733956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:45.733956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:45.733956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 650/1278] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:45.738526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:45.834959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8855eb5-7b09-4be9-b2c5-54880a64b510 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:45.866924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:45.867518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:45.867782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:45.868056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:45.868681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:45.868979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:45.870353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:45.882290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 629/1279] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:45.887882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:45.917888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:45.985328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bfa47-0df7-4ae8-82b9-dc0c7d616b80 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.017512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.018693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.018968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.019235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] PATH_INFO: `/v3/users/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.026346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.027758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 10:51:46.038512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:46.039585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:46.040743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 651/1280] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:46.045466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:46.049258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:46.141357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58217563-c5c5-479f-b8ac-210eac722c51 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.174483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.175079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.175357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.175634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.176348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:46.176653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.178098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.194974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 630/1281] 10.222.0.46 () {56 vars in 1143 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:46.200621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:46.288608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:46.298332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fd07f5-ed02-49e7-88e7-da4b48f615fe None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.331178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.331778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.332055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.332326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.396406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.397785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.480977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:46.492175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 652/1282] 10.222.0.46 () {58 vars in 1282 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 10:51:46.497954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:46.557658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:46.593880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9465db-7a5a-4655-8e9a-493e3311bf6a None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.627352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.627974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.628252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.628520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.638133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.639567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 10:51:46.653549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:46.654655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 10:51:46.655532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 10:51:46.655532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 10:51:46.655532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:46.655532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 631/1283] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:46.660650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:46.755711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6131938d-4c33-43a7-94c2-0e83ee58722a None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.787896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.788489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.788764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.789031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.789656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:46.789958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.791323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.802567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 653/1284] 10.222.0.46 () {56 vars in 1109 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:46.808137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:46.904912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357fda9-d863-4346-91d9-397503fa7169 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:46.918428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:46.940626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:46.941242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:46.941523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:46.941794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:46.948329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:46.949756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 10:51:46.961272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:46.962332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: ERROR keystone.server.flask.application Oct 28 10:51:46.963236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 632/1285] 10.222.0.46 () {56 vars in 1111 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:46.968234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:47.016888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:47.067053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecda6b-5ac4-4d8b-af6a-17ab4ff9b7dd None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:47.098976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:47.099569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:47.099845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:47.100112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:47.100783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:47.101081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:47.102524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba414e-eeb0-4757-ae2e-83f495f2b58d None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:47.114788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 654/1286] 10.222.0.46 () {56 vars in 1148 bytes} [Fri Oct 28 10:51:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:47.120773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:47.215791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c6ce6-9e85-4cb9-94d6-e1127544d1fc None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:47.247614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:47.248199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:47.248471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:47.248737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:47.249451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] RBAC: Adding query filter params (role.id=219649f17b76446aa279da8fc7dc40e7, scope.project.id=f6834f875cac433ca5419f5b85701542, user.id=73aefca6b8d04a0e8bd2fad4ffedbccd) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:47.249753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:47.251208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67104a8-73e2-4656-b1ff-29809d29a199 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:47.257459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 633/1287] 10.222.0.46 () {56 vars in 1374 bytes} [Fri Oct 28 10:51:47 2022] GET /identity/v3/role_assignments?user.id=73aefca6b8d04a0e8bd2fad4ffedbccd&scope.project.id=f6834f875cac433ca5419f5b85701542&role.id=219649f17b76446aa279da8fc7dc40e7 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:49.370279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5444f5d-6ade-489d-b0fa-37e76b680f1e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:49.370791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5444f5d-6ade-489d-b0fa-37e76b680f1e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:49.370899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5444f5d-6ade-489d-b0fa-37e76b680f1e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:49.372143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 655/1288] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:49.377341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:49.377771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:49.377895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:49.396157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:49.397342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:49.426721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:49.462974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 634/1289] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:49.726713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5444f5d-6ade-489d-b0fa-37e76b680f1e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:49.826247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5444f5d-6ade-489d-b0fa-37e76b680f1e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:49.837063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:49.860206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:49.860824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:49.861104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:49.861400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:49.862003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:49.863390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:49.872026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:49.873114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:49.874152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 656/1290] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:51:49 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:49.879675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:49.978937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982fea43-ae9e-4c93-af12-a18c75b0b768 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:50.013796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:50.014429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:50.014711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:50.014987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:50.015617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:50.015920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:50.017292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f81893-d7ff-4ee2-9a53-d4e47ac82c0d None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:50.021958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 635/1291] 10.222.0.46 () {56 vars in 1116 bytes} [Fri Oct 28 10:51:49 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:50.027718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:50.123671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519d5270-4507-45cd-8b2f-08d732235e86 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:50.156394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:50.156981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:50.157260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:50.157532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] PATH_INFO: `/v3/services` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:50.158178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:50.158480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:50.159839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff0ac3d-56aa-4528-acc1-b97ed53ea469 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:50.164154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 657/1292] 10.222.0.46 () {56 vars in 1116 bytes} [Fri Oct 28 10:51:50 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:52.262535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b46ae-22c8-4c15-8f34-1bb3e1c98612 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:52.263077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b46ae-22c8-4c15-8f34-1bb3e1c98612 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:52.263077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82b46ae-22c8-4c15-8f34-1bb3e1c98612 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:52.264406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 636/1293] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:52.270099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db205c7-fb7b-4112-acfb-3a7935fb5046 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:52.270560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db205c7-fb7b-4112-acfb-3a7935fb5046 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:52.270656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db205c7-fb7b-4112-acfb-3a7935fb5046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:52.289013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5db205c7-fb7b-4112-acfb-3a7935fb5046 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:52.290230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5db205c7-fb7b-4112-acfb-3a7935fb5046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:52.318210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:52.354701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 658/1294] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:52.624446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82b46ae-22c8-4c15-8f34-1bb3e1c98612 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:52.654184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:52.722738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82b46ae-22c8-4c15-8f34-1bb3e1c98612 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:52.755064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:52.755644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:52.755918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:52.756182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:52.756820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:52.758188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6808b6-b3cb-4faa-a814-e04c30324a0a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:52.771411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 637/1295] 10.222.0.46 () {60 vars in 1141 bytes} [Fri Oct 28 10:51:52 2022] POST /identity/v3/services => generated 246 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:51:54.864186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93d5bc-c8c6-46b4-bafe-315fbd35b8b0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:54.864740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93d5bc-c8c6-46b4-bafe-315fbd35b8b0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:54.864740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93d5bc-c8c6-46b4-bafe-315fbd35b8b0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:54.866116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 659/1296] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:51:54.871455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:54.871848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:54.872021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:54.890521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:54.891708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:54.969204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 638/1297] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:54 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:55.249801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93d5bc-c8c6-46b4-bafe-315fbd35b8b0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:55.348582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93d5bc-c8c6-46b4-bafe-315fbd35b8b0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:55.381731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:55.382388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:55.382669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:55.382942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] PATH_INFO: `/v3/services/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:55.383518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:55.384887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:55.394016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:55.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:55.396318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 660/1298] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:51:55 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:55.401206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:55.419491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:55.498623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca24517-4dee-485b-9e48-40032e7af031 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:55.530998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:55.531537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:55.531810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:55.532077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:55.532691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:55.532988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:55.534888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:55.539760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 639/1299] 10.222.0.46 () {56 vars in 1116 bytes} [Fri Oct 28 10:51:55 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:55.545662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:55.576537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:55.641656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29affaa-93cf-4e3b-b757-e4cc0d01d21f None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:55.673278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:55.673879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:55.674178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:55.674448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:55.675129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] RBAC: Adding query filter params (interface=public, service_id=6a6dccf7121241eb8ba87003034cdb51, region_id=RegionOne) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:55.675430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:55.676766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962a0640-51dd-4f39-8597-e10176a3715e None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:55.681542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 661/1300] 10.222.0.46 () {56 vars in 1251 bytes} [Fri Oct 28 10:51:55 2022] GET /identity/v3/endpoints?service_id=6a6dccf7121241eb8ba87003034cdb51&interface=public®ion_id=RegionOne => generated 194 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:55.687111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:55.765315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:55.784200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e158cb-d805-4ee7-a13c-b946d36ed762 None admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:55.816846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:55.817435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:55.817714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:55.818003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:55.818605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:55.819953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe4822-2b2d-4b3f-aa7f-e985c77d45ba None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:55.826563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 640/1301] 10.222.0.46 () {56 vars in 1087 bytes} [Fri Oct 28 10:51:55 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 10:51:57.932249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5141c9c-9129-45dc-9301-c95b7baca62a None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:57.932830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5141c9c-9129-45dc-9301-c95b7baca62a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:57.933011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5141c9c-9129-45dc-9301-c95b7baca62a None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:57.934862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 662/1302] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:51:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:51:57.940654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:57.941219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:57.941219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:57.961552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:51:57.963082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:51:58.029010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 641/1303] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:51:57 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 10:51:58.297836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5141c9c-9129-45dc-9301-c95b7baca62a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:58.389801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:58.396827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5141c9c-9129-45dc-9301-c95b7baca62a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:58.428991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:58.429611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:58.429886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:58.430185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:58.430755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:58.432101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 10:51:58.441172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 10:51:58.442338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application with Lock( Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self._enter() Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application return self.creator() Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application created_value = creator( Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: ERROR keystone.server.flask.application Oct 28 10:51:58.443369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 663/1304] 10.222.0.46 () {56 vars in 1117 bytes} [Fri Oct 28 10:51:58 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 10:51:58.448248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:58.521373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:51:58.544373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d07a8-acb6-4328-ae13-cfc5b59f55ab None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:58.576934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:58.577537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:58.577817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:58.578119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] PATH_INFO: `/v3/services` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:58.578744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:51:58.579048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:58.580441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b0b604-2487-4b0a-8e6c-7874eb214c8a None admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:58.585388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 642/1305] 10.222.0.46 () {56 vars in 1116 bytes} [Fri Oct 28 10:51:58 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:51:58.591140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:51:58.686993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bc155-777c-4268-9443-0c81274e6862 None admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:51:58.718703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:51:58.719283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:51:58.719557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:51:58.719821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:51:58.720985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:51:58.722408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7b1da4-b844-46b0-ba3f-54ecc47ff7b1 None admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:51:58.741173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 664/1306] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:51:58 2022] POST /identity/v3/endpoints => generated 331 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:52:08.582835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=30 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:52:08.736958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:01.854284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086936e-6c19-4bc1-b9e7-87ae577c6934 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:01.854791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086936e-6c19-4bc1-b9e7-87ae577c6934 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:01.855365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c086936e-6c19-4bc1-b9e7-87ae577c6934 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:01.864007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ccd312-de1e-4a74-a53c-9d28bc95dca3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:01.864504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ccd312-de1e-4a74-a53c-9d28bc95dca3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:01.864682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ccd312-de1e-4a74-a53c-9d28bc95dca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:01.868198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 643/1307] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:01 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:02.023149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f2ccd312-de1e-4a74-a53c-9d28bc95dca3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:02.024792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f2ccd312-de1e-4a74-a53c-9d28bc95dca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:02.152151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 665/1308] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:02.444340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:02.444623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:02.445048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:02.446738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 644/1309] 10.222.0.46 () {54 vars in 964 bytes} [Fri Oct 28 10:53:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:02.453293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1fab6-2c05-4df7-a6f4-052131accdcc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:02.454272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1fab6-2c05-4df7-a6f4-052131accdcc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:02.454460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1fab6-2c05-4df7-a6f4-052131accdcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:02.543774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f9e1fab6-2c05-4df7-a6f4-052131accdcc None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:02.544950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f9e1fab6-2c05-4df7-a6f4-052131accdcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:02.567682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:02.627355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 666/1310] 10.222.0.46 () {58 vars in 1061 bytes} [Fri Oct 28 10:53:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:02.636195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:02.637952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:02.747699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe396b0-7b69-4912-9293-38f05fa9e54a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:02.824024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:02.824640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:02.824919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:02.825193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:02.826846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:02.887945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:02.951153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:02.952620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c182b19-6cf6-4e44-9224-811310fe62d6 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:02.977880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 645/1311] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:05.485718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:05.486377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:05.486377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:05.487628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 667/1312] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:05.492872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d867ece-d4b9-4505-8cf8-72b26caa8f32 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:05.493302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d867ece-d4b9-4505-8cf8-72b26caa8f32 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:05.493411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d867ece-d4b9-4505-8cf8-72b26caa8f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:05.563212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7d867ece-d4b9-4505-8cf8-72b26caa8f32 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:05.564417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7d867ece-d4b9-4505-8cf8-72b26caa8f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:05.659331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 646/1313] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:05.945185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:05.946801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:06.050630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f90e17-36f3-4e8a-b73c-1277012509ab None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:06.121341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:06.123496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:06.124166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:06.124461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:06.124759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:06.126947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:06.232183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:06.233779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898063a6-0948-4ff3-8f4f-af4e75a2af15 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:06.251854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 668/1314] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:08.673903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:08.674454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:08.674575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:08.675847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 647/1315] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:08.681042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a938ff-dbcf-4ee6-be00-ddc80a3b8107 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:08.681422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a938ff-dbcf-4ee6-be00-ddc80a3b8107 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:08.681584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a938ff-dbcf-4ee6-be00-ddc80a3b8107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:08.746056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e7a938ff-dbcf-4ee6-be00-ddc80a3b8107 None None] MFA Rules not processed for user `7d855e0bceeb417d9bfcb6751dd67251`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:08.747246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e7a938ff-dbcf-4ee6-be00-ddc80a3b8107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:08.796457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:08.840096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 669/1316] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:08 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:09.123658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:09.125301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:09.135094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:09.227257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b5d45-badb-4662-8958-56248e7c07ea None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:09.300128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:09.300726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:09.301000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:09.301268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:09.302878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:09.363446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:09.410252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:09.411618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969ea410-3bef-43bf-b19c-f6895bb23ce3 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:09.428922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 648/1317] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:09 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:12.049078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:12.049642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:12.049642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:12.051035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 670/1318] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:12.056148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35667028-32c7-4938-9d83-399e87c1a211 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:12.056566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35667028-32c7-4938-9d83-399e87c1a211 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:12.056691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35667028-32c7-4938-9d83-399e87c1a211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:12.121837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-35667028-32c7-4938-9d83-399e87c1a211 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:12.123061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-35667028-32c7-4938-9d83-399e87c1a211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:12.217833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 649/1319] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:12 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:12.502509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:12.504050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:12.604557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e74a1c-8860-415a-8260-3de42751d385 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:12.677482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:12.678118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:12.678395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:12.678664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:12.680230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:12.686937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:12.788455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:12.789821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b88d0-9b67-42ec-a4a5-5fc688892c04 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:12.807491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 671/1320] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:15.575162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:15.575702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:15.575702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:15.577039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 650/1321] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:15.582377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420ef0f-003a-45ce-9880-0e8aca4abc78 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:15.582801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420ef0f-003a-45ce-9880-0e8aca4abc78 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:15.582940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1420ef0f-003a-45ce-9880-0e8aca4abc78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:15.656699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-1420ef0f-003a-45ce-9880-0e8aca4abc78 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:15.657947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1420ef0f-003a-45ce-9880-0e8aca4abc78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:15.719142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:15.761353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 672/1322] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:16.042374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:16.044010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:16.064438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:16.150234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c315d-9ea2-4966-ad82-a8528439376e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:16.226116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:16.226728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:16.227006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:16.227278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:16.228986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:16.301565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:16.339978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:16.341412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d0ca34-a0d4-4f47-a130-f494b3b7f4fc service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:16.360531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 651/1323] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:18.551448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:18.551967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:18.551967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:18.553245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 673/1324] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:18.558472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3f832-2cfb-428a-b376-ace9f9fdb3e1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:18.558889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3f832-2cfb-428a-b376-ace9f9fdb3e1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:18.559051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3f832-2cfb-428a-b376-ace9f9fdb3e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:18.626689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b5f3f832-2cfb-428a-b376-ace9f9fdb3e1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:18.627911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b5f3f832-2cfb-428a-b376-ace9f9fdb3e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:18.733018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 652/1325] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:19.013510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:19.015067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:19.119341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86e2b6f-19d3-47e2-9b5a-58581e4a8ba8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:19.194321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:19.194931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:19.195214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:19.195484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:19.197078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:19.214597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:19.311761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:19.313332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7878f4-2a41-4f7a-863e-0556c92c4a9a service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:19.332065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 674/1326] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:21.518121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be96f3-a4b7-4607-90b3-fe7495e7445f None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:21.518620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be96f3-a4b7-4607-90b3-fe7495e7445f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:21.518620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be96f3-a4b7-4607-90b3-fe7495e7445f None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:21.519878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 653/1327] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:21.525256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d620784-c733-4690-8c10-58729fcc49d2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:21.525699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d620784-c733-4690-8c10-58729fcc49d2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:21.525801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d620784-c733-4690-8c10-58729fcc49d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:21.596520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4d620784-c733-4690-8c10-58729fcc49d2 None None] MFA Rules not processed for user `7d855e0bceeb417d9bfcb6751dd67251`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:21.597734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4d620784-c733-4690-8c10-58729fcc49d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:21.674293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:21.707781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 675/1328] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:21 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:22.003318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:22.003706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:22.003981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:22.005317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 654/1329] 10.222.0.46 () {54 vars in 964 bytes} [Fri Oct 28 10:53:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:22.011459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152faf20-ecee-4e16-8712-a86c097fc370 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:22.011751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152faf20-ecee-4e16-8712-a86c097fc370 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:22.012033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152faf20-ecee-4e16-8712-a86c097fc370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:22.087522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-152faf20-ecee-4e16-8712-a86c097fc370 None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:22.088646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-152faf20-ecee-4e16-8712-a86c097fc370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:22.164773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 676/1330] 10.222.0.46 () {58 vars in 1061 bytes} [Fri Oct 28 10:53:22 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:53:22.173806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:22.175455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:22.206536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:22.280107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a33d1f2-4d60-4aa8-8ff7-c2ccd6c33d39 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:22.356150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:22.356672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:22.356947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:22.357215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:22.359091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:22.436708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:22.471357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:22.472837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c43f1a-8a45-4cac-9513-da4666ff84da service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:22.490919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 655/1331] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:22 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:25.225952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:25.226576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:25.226576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:25.227863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 677/1332] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:25.232970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b169-2ba5-45c4-95e7-55bf61d8d029 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:25.233417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b169-2ba5-45c4-95e7-55bf61d8d029 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:25.233783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b169-2ba5-45c4-95e7-55bf61d8d029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:25.299328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c414b169-2ba5-45c4-95e7-55bf61d8d029 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:25.300523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c414b169-2ba5-45c4-95e7-55bf61d8d029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:25.395143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 656/1333] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:25.683010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:25.684575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:25.754846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:25.788704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e578a672-0f08-4bcb-af76-cfc6fb8bfb92 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:25.859305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:25.859902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:25.860173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:25.860436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:25.862017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:25.969292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:25.970679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9246f1c-f74d-4319-a105-b0533e095f0b service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:25.980614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:25.989530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 678/1334] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:28.146658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:28.147271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:28.147271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:28.148550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 657/1335] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:28.153928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d63bd-bf07-4126-8191-00d795b70e41 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:28.154397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d63bd-bf07-4126-8191-00d795b70e41 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:28.154517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566d63bd-bf07-4126-8191-00d795b70e41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:28.219018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-566d63bd-bf07-4126-8191-00d795b70e41 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:28.220214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-566d63bd-bf07-4126-8191-00d795b70e41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:28.315949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 679/1336] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:28.601084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:28.602658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:28.643948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:28.707264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130e5971-3bf6-4ac8-b815-893d09b58218 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:28.780646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:28.781753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:28.782057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:28.782325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:28.783908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:28.864338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:28.889168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:28.891309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2536d3-fd59-42ea-adbd-e9d38f9eb8c2 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:28.909800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 658/1337] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:31.479538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:31.480090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:31.480090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:31.481408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 680/1338] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:31.486748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd21c43-e34e-4dd6-a6a1-2fe1d862adbd None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:31.487167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd21c43-e34e-4dd6-a6a1-2fe1d862adbd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:31.487349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd21c43-e34e-4dd6-a6a1-2fe1d862adbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:31.551577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2fd21c43-e34e-4dd6-a6a1-2fe1d862adbd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:31.552711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2fd21c43-e34e-4dd6-a6a1-2fe1d862adbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:31.647389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 659/1339] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:53:31.925870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:31.927427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:32.011931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:32.033477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85878d-6a7c-4562-bc7a-75a8a029f3aa None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:32.106415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:32.107015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:32.107301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:32.107552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:32.109884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:32.215934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:32.217289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc04329-6517-44f7-a5f6-edf003412edc service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:32.235685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 681/1340] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:36.379525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:36.380080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:36.380208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:36.381455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 660/1341] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:36.386808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a1f07-ee44-4c57-9eff-4313268c5fc5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:36.387218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a1f07-ee44-4c57-9eff-4313268c5fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:36.387411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09a1f07-ee44-4c57-9eff-4313268c5fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:36.393324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:36.456754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d09a1f07-ee44-4c57-9eff-4313268c5fc5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:36.458244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d09a1f07-ee44-4c57-9eff-4313268c5fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:36.553228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 682/1342] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:36.835733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:36.837326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:36.887118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:36.943131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48515d51-aa6b-4e2e-9314-57c66a6a93eb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:37.017850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:37.018494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:37.018772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:37.019045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:37.020646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:37.115297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:37.131891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:37.133402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f785c2-52ca-46d7-8f9b-d42448e951f4 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:37.151800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 661/1343] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:39.921648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:39.922244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:39.922349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:39.923677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 683/1344] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:39.928933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12da7bab-3fce-4f90-95fb-57ecd461abf8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:39.929363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12da7bab-3fce-4f90-95fb-57ecd461abf8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:39.929617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12da7bab-3fce-4f90-95fb-57ecd461abf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:40.003873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-12da7bab-3fce-4f90-95fb-57ecd461abf8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:40.005082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-12da7bab-3fce-4f90-95fb-57ecd461abf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:40.100745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 662/1345] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:39 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:40.381112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:40.382751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:40.476209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:40.493905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b7bce4-0533-4878-9eb9-af71a1f32dac None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:40.569900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:40.570457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:40.570737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:40.571010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:40.572628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:40.681463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:40.682947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af0ff23-e72f-4eaf-9c25-f47e1c5fd1b7 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:40.701882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 684/1346] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:40 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:45.067909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:45.068450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:45.068450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:45.069846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 663/1347] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:53:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:45.075418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6a527-4289-4085-be4d-38630ff06f11 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:45.075873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6a527-4289-4085-be4d-38630ff06f11 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:45.075970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6a527-4289-4085-be4d-38630ff06f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:45.091876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:45.146869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b0b6a527-4289-4085-be4d-38630ff06f11 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:45.148078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b0b6a527-4289-4085-be4d-38630ff06f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:45.245692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 685/1348] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:53:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:45.534153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:53:45.535701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:45.597868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:45.645108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e06de-e566-4772-abe1-39c5113d1d1e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:53:45.716225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:53:45.716825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:45.717096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:45.717360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:45.719020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:45.823386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:45.826634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:53:45.828572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c8362-09ef-42b2-96d1-42781302d450 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:53:45.846907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 664/1349] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:53:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:53:50.693441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0aea61-8133-4dd9-9fd8-172471b96e20 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:50.694430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0aea61-8133-4dd9-9fd8-172471b96e20 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:50.695950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0aea61-8133-4dd9-9fd8-172471b96e20 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:50.697786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 686/1350] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 10:53:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:50.703686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10f21a-855f-4e7a-b657-cead10dda2ec None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:50.704132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10f21a-855f-4e7a-b657-cead10dda2ec None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:50.704255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10f21a-855f-4e7a-b657-cead10dda2ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:50.871577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8a10f21a-855f-4e7a-b657-cead10dda2ec None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:50.873428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8a10f21a-855f-4e7a-b657-cead10dda2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:51.003130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 665/1351] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 10:53:50 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:55.062927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d39ff-bfd5-4ba6-a043-1cfc8267df04 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:55.063877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d39ff-bfd5-4ba6-a043-1cfc8267df04 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:55.064306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d39ff-bfd5-4ba6-a043-1cfc8267df04 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:55.066017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 687/1352] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 10:53:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:55.071790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6309095-f116-44c4-a9b2-0e268a7ad574 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:55.072204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6309095-f116-44c4-a9b2-0e268a7ad574 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:55.072509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6309095-f116-44c4-a9b2-0e268a7ad574 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:55.134077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:55.160104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b6309095-f116-44c4-a9b2-0e268a7ad574 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:55.161608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b6309095-f116-44c4-a9b2-0e268a7ad574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:55.244279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=59 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:55.282122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 666/1353] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 10:53:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:53:56.611530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db329a8-c745-4f4a-b5dc-8c6df503b23a None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:56.612078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db329a8-c745-4f4a-b5dc-8c6df503b23a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:56.612078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db329a8-c745-4f4a-b5dc-8c6df503b23a None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:56.613397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 688/1354] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 10:53:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:53:56.618510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305f968-c5f3-453c-9d19-0099ed7a8ad8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:53:56.618899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305f968-c5f3-453c-9d19-0099ed7a8ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:53:56.619069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305f968-c5f3-453c-9d19-0099ed7a8ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:53:56.697293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2305f968-c5f3-453c-9d19-0099ed7a8ad8 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:53:56.698522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2305f968-c5f3-453c-9d19-0099ed7a8ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:53:56.740381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:53:56.805383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 667/1355] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 10:53:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:00.406874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844a250-5c0b-4bff-a941-e25e5a931fe6 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:00.407422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844a250-5c0b-4bff-a941-e25e5a931fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:00.407585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844a250-5c0b-4bff-a941-e25e5a931fe6 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:00.409028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 689/1356] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:00.414603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd1a14-32c6-43f1-a303-d870b36dfa51 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:00.415088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd1a14-32c6-43f1-a303-d870b36dfa51 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:00.415388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd1a14-32c6-43f1-a303-d870b36dfa51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:00.489729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3fbd1a14-32c6-43f1-a303-d870b36dfa51 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:00.491236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3fbd1a14-32c6-43f1-a303-d870b36dfa51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:00.594137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 668/1357] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:00.900729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:00.901550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:00.902305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:00.904471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 690/1358] 10.222.0.46 () {54 vars in 960 bytes} [Fri Oct 28 10:54:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:00.912417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba6200-9334-472e-b714-6ee53721c7c1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:00.912713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba6200-9334-472e-b714-6ee53721c7c1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:00.912993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ba6200-9334-472e-b714-6ee53721c7c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:00.933639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:01.002785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-53ba6200-9334-472e-b714-6ee53721c7c1 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:01.004123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-53ba6200-9334-472e-b714-6ee53721c7c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:01.136854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 669/1359] 10.222.0.46 () {58 vars in 1057 bytes} [Fri Oct 28 10:54:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:01.145293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:01.146942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:01.314002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a5c51b-fc19-4211-b4bd-bc883427769f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:01.391529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:01.434272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:01.434989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:01.435301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:01.435629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:01.437816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:01.538813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:01.540482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfe124c-dcb7-41dd-b744-327cc3749783 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:01.559934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 691/1360] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:03.935489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:03.936021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:03.936021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:03.937435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 670/1361] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:03.943030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be745728-25ad-43ec-aef3-ecd453a8a0fc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:03.943498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be745728-25ad-43ec-aef3-ecd453a8a0fc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:03.944171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be745728-25ad-43ec-aef3-ecd453a8a0fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:04.047103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-be745728-25ad-43ec-aef3-ecd453a8a0fc None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:04.048304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-be745728-25ad-43ec-aef3-ecd453a8a0fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:04.088486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:04.151585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 692/1362] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:54:04.440788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:04.442359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:04.509039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:04.613355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2893bf2e-a24e-46e1-ba68-2b81d80ce6a0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:04.744444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:04.745385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:04.745385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:04.745590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:04.747368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:04.772178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:04.842662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:04.844543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6429e0-4d33-472a-aa75-9df389410c06 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:04.865458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 671/1363] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:07.363128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7aec2d-966f-4c4b-aeb4-3a13978c3903 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:07.363627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7aec2d-966f-4c4b-aeb4-3a13978c3903 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:07.363705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7aec2d-966f-4c4b-aeb4-3a13978c3903 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:07.365080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 693/1364] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:07.370298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29132b23-5b09-4834-8a27-4043400cd8f4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:07.370721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29132b23-5b09-4834-8a27-4043400cd8f4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:07.370887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29132b23-5b09-4834-8a27-4043400cd8f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:07.439357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-29132b23-5b09-4834-8a27-4043400cd8f4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:07.440564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-29132b23-5b09-4834-8a27-4043400cd8f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:07.543847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:07.554696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 672/1365] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:07.839291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:07.839596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:07.839840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:07.841285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 694/1366] 10.222.0.46 () {54 vars in 960 bytes} [Fri Oct 28 10:54:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:07.846895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbec39-4deb-426a-b9ab-cec287ecd9b3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:07.847285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbec39-4deb-426a-b9ab-cec287ecd9b3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:07.847679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dbec39-4deb-426a-b9ab-cec287ecd9b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:07.927938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-15dbec39-4deb-426a-b9ab-cec287ecd9b3 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:07.929311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-15dbec39-4deb-426a-b9ab-cec287ecd9b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:08.052889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 673/1367] 10.222.0.46 () {58 vars in 1057 bytes} [Fri Oct 28 10:54:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:08.060605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:08.062959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:08.219864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae4de5-4653-40d7-ba55-a5ba6d1bc245 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:08.241248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:08.345262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:08.345879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:08.346192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:08.346467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:08.348064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:08.443227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:08.444972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15a0cff-84d2-46f9-9a77-e43bdbcfb21f service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:08.466024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 695/1368] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:08.588211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fee37f-f928-4cad-a2ec-0bb437108798 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:08.588769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fee37f-f928-4cad-a2ec-0bb437108798 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:08.589205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fee37f-f928-4cad-a2ec-0bb437108798 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:08.590993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 674/1369] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:08.597145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2598e8-2c87-47c5-b7a9-ccd5499ebf45 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:08.597479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2598e8-2c87-47c5-b7a9-ccd5499ebf45 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:08.597726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2598e8-2c87-47c5-b7a9-ccd5499ebf45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:08.636262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:08.683032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-df2598e8-2c87-47c5-b7a9-ccd5499ebf45 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:08.684334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-df2598e8-2c87-47c5-b7a9-ccd5499ebf45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:08.805228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 696/1370] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:09.143654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:09.143654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:09.143654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:09.145565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 675/1371] 10.222.0.46 () {54 vars in 957 bytes} [Fri Oct 28 10:54:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:09.149048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b65b12-874b-4659-ae1e-0822fda37d55 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:09.149334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b65b12-874b-4659-ae1e-0822fda37d55 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:09.149605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b65b12-874b-4659-ae1e-0822fda37d55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:09.231331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-67b65b12-874b-4659-ae1e-0822fda37d55 None None] MFA Rules not processed for user `2297b30fe14d4de8a3f39e247ed59dfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:09.232891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-67b65b12-874b-4659-ae1e-0822fda37d55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:09.290805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:09.323437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 697/1372] 10.222.0.46 () {58 vars in 1054 bytes} [Fri Oct 28 10:54:09 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:09.331610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:09.333566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:09.388417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:09.456421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bce23d-e12d-4e0f-80c8-7c20b1932d7b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:09.547660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:09.548340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:09.548626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:09.548899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:09.550749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:09.669200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:09.682791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:09.684122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb997fb-f07d-41fa-aa69-21400c59672a service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:09.703636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 676/1373] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:10.698240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:10.698783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:10.698783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:10.700128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 698/1374] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:10.705513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869c187-3cdf-47ee-bca6-a5a77bd0bd7e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:10.705925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869c187-3cdf-47ee-bca6-a5a77bd0bd7e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:10.706089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869c187-3cdf-47ee-bca6-a5a77bd0bd7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:10.771780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a869c187-3cdf-47ee-bca6-a5a77bd0bd7e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:10.772985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a869c187-3cdf-47ee-bca6-a5a77bd0bd7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:10.870328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 677/1375] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:11.164820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:11.166400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:11.328063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe91695-3717-4c60-92aa-177721c18d33 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:11.384957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:11.447845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:11.448463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:11.448763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:11.449020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:11.450637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:11.545329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:11.547161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25257009-c0f9-4582-9918-f8807e8c7d60 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:11.567677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 699/1376] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:13.810429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:13.810979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:13.811229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:13.812867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 678/1377] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:13.818079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80adcd6-3180-4b58-8455-826fd1013240 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:13.818507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80adcd6-3180-4b58-8455-826fd1013240 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:13.818679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80adcd6-3180-4b58-8455-826fd1013240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:13.890294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a80adcd6-3180-4b58-8455-826fd1013240 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:13.891662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a80adcd6-3180-4b58-8455-826fd1013240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:13.899325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:14.001431 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 700/1378] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:14.292621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:14.294177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:14.356080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:14.447826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb169ea0-c869-46ff-ae97-278317d62a8a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:14.563876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:14.564401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:14.564674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:14.564938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:14.566580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:14.586039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:14.662248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:14.663839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984af9d4-08be-4c2e-be07-4971c9e1c679 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:14.685327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 679/1379] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:16.928414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:16.928858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:16.929263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:16.931061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 701/1380] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:16.936983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ccfb02-fb8f-4f72-a481-176d6ea523ca None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:16.937411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ccfb02-fb8f-4f72-a481-176d6ea523ca None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:16.937568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ccfb02-fb8f-4f72-a481-176d6ea523ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:17.013865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-96ccfb02-fb8f-4f72-a481-176d6ea523ca None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:17.015854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-96ccfb02-fb8f-4f72-a481-176d6ea523ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:17.150757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:17.157748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 680/1381] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:17.453506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:17.455061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:17.603221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:17.615006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413598be-4e9b-42e0-9e21-111cd5ec13e7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:17.731018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:17.731714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:17.732085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:17.732443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:17.734990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:17.845282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:17.846931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eac73b7-ba75-49cc-8b9e-502dea4e45d7 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:17.857028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:17.867415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 702/1382] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:20.058067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:20.058606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:20.058606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:20.060013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 681/1383] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:20.066365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20b5c4-ec9f-40d7-b913-c2a4be9c808f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:20.066903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20b5c4-ec9f-40d7-b913-c2a4be9c808f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:20.066903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc20b5c4-ec9f-40d7-b913-c2a4be9c808f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:20.145489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-fc20b5c4-ec9f-40d7-b913-c2a4be9c808f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:20.146615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fc20b5c4-ec9f-40d7-b913-c2a4be9c808f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:20.245260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 703/1384] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:20.546249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:20.547857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:20.703003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762fbb26-f886-49e7-8f75-dbc05c287cc7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:20.773028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:20.825497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:20.826365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:20.827325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:20.827667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:20.829777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:20.935750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:20.937389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4955e161-442d-45cf-93ec-9a1fd6a33749 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:20.959140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 682/1385] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 10:54:23.263270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:23.263781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:23.263781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:23.265793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 704/1386] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:23.270936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed386d-6769-42d7-a760-fd03f26088af None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:23.271361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed386d-6769-42d7-a760-fd03f26088af None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:23.271544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ed386d-6769-42d7-a760-fd03f26088af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:23.338170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-04ed386d-6769-42d7-a760-fd03f26088af None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:23.339735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-04ed386d-6769-42d7-a760-fd03f26088af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:23.372865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:23.436812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 683/1387] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:23.737928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:23.739465 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:23.815170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:23.884569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5decc2-52f2-4ea9-bf7e-5c4c3ee14710 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:23.999098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:23.999631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:23.999912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:24.000187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:24.001853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:24.041068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:24.092109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:24.093669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74538d6c-af80-4e9b-831d-4ea29b469595 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:24.111891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 705/1388] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:24.901544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ac1a0-9ff5-44fc-ab33-27c50df7614e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:24.902152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ac1a0-9ff5-44fc-ab33-27c50df7614e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:24.902152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ac1a0-9ff5-44fc-ab33-27c50df7614e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:24.903579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 684/1389] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:24.908878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49523ca7-1200-44db-b4cf-3fbe890c7aac None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:24.909335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49523ca7-1200-44db-b4cf-3fbe890c7aac None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:24.909443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49523ca7-1200-44db-b4cf-3fbe890c7aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:24.976259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-49523ca7-1200-44db-b4cf-3fbe890c7aac None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:24.977472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-49523ca7-1200-44db-b4cf-3fbe890c7aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:25.075961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 706/1390] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:25.378203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:25.378574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:25.378911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:25.380349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 685/1391] 10.222.0.46 () {54 vars in 962 bytes} [Fri Oct 28 10:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:25.385673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaac1e3-35a0-4fc5-b1d2-5bb6d55ef4d8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:25.385960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaac1e3-35a0-4fc5-b1d2-5bb6d55ef4d8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:25.386264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaac1e3-35a0-4fc5-b1d2-5bb6d55ef4d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:25.402547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:25.464798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-edaac1e3-35a0-4fc5-b1d2-5bb6d55ef4d8 None None] MFA Rules not processed for user `589508bdaf3c4e159c01e3a43eb69392`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:25.466052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-edaac1e3-35a0-4fc5-b1d2-5bb6d55ef4d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:25.542103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 707/1392] 10.222.0.46 () {58 vars in 1059 bytes} [Fri Oct 28 10:54:25 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:25.551175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:25.552675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:25.662468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0603290-6063-407e-8ee8-625dbdd4695c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:25.721046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:25.737540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:25.738468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:25.738746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:25.739015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:25.740616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:25.854660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:25.856115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1aa53-584b-4310-99ef-d15ba56e876f service glance] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:25.875337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 686/1393] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:26.273253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d39e8-e5d9-4077-96c5-feb01d0ead8f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.273725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d39e8-e5d9-4077-96c5-feb01d0ead8f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.273842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d39e8-e5d9-4077-96c5-feb01d0ead8f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.275214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 708/1394] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:26.283240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.283790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.283790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.349079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:26.350329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:26.395622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:26.454870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 687/1395] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:26.624682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a046452-76f4-4c66-ac5d-f59776fab784 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.625002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a046452-76f4-4c66-ac5d-f59776fab784 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.625281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a046452-76f4-4c66-ac5d-f59776fab784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.692650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9a046452-76f4-4c66-ac5d-f59776fab784 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:26.693832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9a046452-76f4-4c66-ac5d-f59776fab784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:26.752153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:26.752323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:26.753882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:26.786134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 709/1396] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 10:54:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:26.834044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fa050-e2b4-423c-a578-02eedcd68642 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.834337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fa050-e2b4-423c-a578-02eedcd68642 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.834619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fa050-e2b4-423c-a578-02eedcd68642 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.835983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 710/1397] 10.222.0.46 () {54 vars in 943 bytes} [Fri Oct 28 10:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:54:26.842869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.843139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.843548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.874739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e270cfab-1542-4ff5-b0a5-5c4843dd3af6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:26.920035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] MFA Rules not processed for user `0449fafc849145c8b752a0b028ad2679`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:26.921767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:26.931074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:26.991764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:26.992377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:26.992919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:26.993197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:26.994839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:27.034421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 711/1398] 10.222.0.46 () {58 vars in 1040 bytes} [Fri Oct 28 10:54:26 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:54:27.043893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:27.053711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de3b63-13a9-444a-8950-8e0f559e7f4f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:27.078987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:27.088221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:27.089953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d19a365-a2aa-4bde-9890-1860ccef776d service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:27.108884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 688/1399] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 10:54:27.160521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:27.161132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:27.161447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:27.161678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:27.163835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:27.263951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:27.265581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af833f70-01c3-483e-b8d2-abfa1bda4cf5 service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:27.275028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 712/1400] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 10:54:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 10:54:29.302052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:29.302600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:29.302600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:29.304006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 689/1401] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:29.309249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f533ebf-e63e-4b90-a03a-de8ba162efb5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:29.309701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f533ebf-e63e-4b90-a03a-de8ba162efb5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:29.309813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f533ebf-e63e-4b90-a03a-de8ba162efb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:29.379485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0f533ebf-e63e-4b90-a03a-de8ba162efb5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:29.380691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0f533ebf-e63e-4b90-a03a-de8ba162efb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:29.389812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:29.493658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 713/1402] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:29.779934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:29.781504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:29.855772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:29.942360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b395f7f-ec0f-4558-9b82-eb80fbafa502 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:30.071014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:30.071635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:30.071909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:30.072179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:30.073818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:30.101948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:30.174026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:30.175635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5441e20b-5404-41b3-bd84-f41a5364cb52 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:30.195668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 690/1403] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:32.375697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:32.376250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:32.376250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:32.377716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 714/1404] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:32.383100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83dfd43-34b0-47a0-9ba7-e62cf6d56005 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:32.383582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83dfd43-34b0-47a0-9ba7-e62cf6d56005 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:32.383702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83dfd43-34b0-47a0-9ba7-e62cf6d56005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:32.463670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d83dfd43-34b0-47a0-9ba7-e62cf6d56005 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:32.464962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d83dfd43-34b0-47a0-9ba7-e62cf6d56005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:32.622138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 691/1405] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:32.903357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:32.905577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:33.057536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:33.069029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b812e4e7-0ded-42b6-a29f-9b3ce81326c2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:33.192910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:33.193468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:33.193751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:33.194054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:33.195961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:33.296492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:33.298116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8472ea-2e88-4d5b-b532-ee65e8d631de service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:33.309693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:33.320908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 715/1406] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:35.516417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:35.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:35.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:35.521533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 692/1407] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:54:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:35.523692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80d6b2-fd12-4183-9c50-d5e09f9c030f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:35.524091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80d6b2-fd12-4183-9c50-d5e09f9c030f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:35.524185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c80d6b2-fd12-4183-9c50-d5e09f9c030f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:35.596065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0c80d6b2-fd12-4183-9c50-d5e09f9c030f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:35.597297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0c80d6b2-fd12-4183-9c50-d5e09f9c030f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:35.707733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 716/1408] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:35.997379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:35.999833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:36.002977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:36.172258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fef1b1-4e49-4c5d-8bc5-6f4cc6a6ddbd None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:36.257719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:36.303544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:36.304154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:36.304435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:36.304702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:36.306352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:36.413833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:36.415657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf1cb83-a6a7-46dd-897c-12887baeb70e service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:36.435530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 693/1409] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:45.706203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=67 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:46.431705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=61 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:49.913936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5b23b-d7d6-4e01-84a6-73562f0b2308 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:49.914574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5b23b-d7d6-4e01-84a6-73562f0b2308 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:49.914574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5b23b-d7d6-4e01-84a6-73562f0b2308 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:49.915845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 717/1410] 10.222.0.46 () {54 vars in 918 bytes} [Fri Oct 28 10:54:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:49.921475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af7362-7e9c-4ce3-a284-b62ea3a41f18 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:49.921859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af7362-7e9c-4ce3-a284-b62ea3a41f18 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:49.922035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af7362-7e9c-4ce3-a284-b62ea3a41f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:49.996501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-32af7362-7e9c-4ce3-a284-b62ea3a41f18 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:49.998335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-32af7362-7e9c-4ce3-a284-b62ea3a41f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:50.103848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 694/1411] 10.222.0.46 () {58 vars in 1015 bytes} [Fri Oct 28 10:54:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:50.118533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5963dc-d420-4555-877b-331f94e800de None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:50.118822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5963dc-d420-4555-877b-331f94e800de None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:50.119100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5963dc-d420-4555-877b-331f94e800de None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:50.120432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 718/1412] 10.222.0.46 () {54 vars in 918 bytes} [Fri Oct 28 10:54:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:50.125067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b709a9fc-5db8-43b5-a7f8-a6d784467ad6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:50.125350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b709a9fc-5db8-43b5-a7f8-a6d784467ad6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:50.125622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b709a9fc-5db8-43b5-a7f8-a6d784467ad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:50.197061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:50.201027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b709a9fc-5db8-43b5-a7f8-a6d784467ad6 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:50.202610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b709a9fc-5db8-43b5-a7f8-a6d784467ad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:50.318448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 695/1413] 10.222.0.46 () {58 vars in 1015 bytes} [Fri Oct 28 10:54:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:52.881040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:52.881614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:52.881614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:52.883025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 719/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:52.888961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b7ef14-388a-47dd-bead-05bac0724279 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:52.889381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b7ef14-388a-47dd-bead-05bac0724279 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:52.889508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b7ef14-388a-47dd-bead-05bac0724279 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:52.956203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-46b7ef14-388a-47dd-bead-05bac0724279 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:52.957402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-46b7ef14-388a-47dd-bead-05bac0724279 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:53.011449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:53.052908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 696/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:54:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:53.341395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:53.343051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:53.487743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fbf6a4-c740-4709-8f3e-b8cf2127dd9c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:53.559138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:53.598440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:53.599033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:53.599331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:53.599599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:53.601161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:53.688014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:53.689603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5593ea-ea7b-4d34-b8fa-d573f8df9aa6 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:53.707595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 720/1416] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:54.840342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:54.841414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:54.841414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:54.842768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 697/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:54:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:54.848787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceed8504-3db3-4d64-858d-0a7f59a0175c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:54.849217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceed8504-3db3-4d64-858d-0a7f59a0175c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:54.849317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceed8504-3db3-4d64-858d-0a7f59a0175c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:54.924175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ceed8504-3db3-4d64-858d-0a7f59a0175c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:54.925376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ceed8504-3db3-4d64-858d-0a7f59a0175c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:54.952843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:55.021539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 721/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:54:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:55.170328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:55.171774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:55.314627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db530c0f-55b1-4abc-980b-96f67e77b047 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:55.353525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:55.425956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:55.426519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:55.426794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:55.427065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:55.428661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:55.515798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:55.517372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb236a-9e79-44a1-9338-0cbadd6acdac service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:55.535235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 698/1419] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:56.663628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:56.664196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:56.664196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:56.665513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 722/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:54:56.672232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a940d-8e61-405b-a465-556cb409e70a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:56.672751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a940d-8e61-405b-a465-556cb409e70a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:56.672848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a940d-8e61-405b-a465-556cb409e70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:56.726184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:56.748566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a09a940d-8e61-405b-a465-556cb409e70a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:56.749758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a09a940d-8e61-405b-a465-556cb409e70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:56.844582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 699/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:54:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:56.995402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:56.997460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:57.141775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723e616b-e806-4d4d-b2d9-5ccb911fec06 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:57.154184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:57.253225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:57.253748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:57.254050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:57.254319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:57.255882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:57.344671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:57.346263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6494827e-c446-4a19-a1d1-bec6be2941f9 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:57.364720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 723/1422] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:54:58.469077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:58.469606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:58.469606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:58.470955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 700/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:54:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:54:58.477016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786bedf8-bd2f-4b4c-936a-ea63324b23cd None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:58.477436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786bedf8-bd2f-4b4c-936a-ea63324b23cd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:58.477560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786bedf8-bd2f-4b4c-936a-ea63324b23cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:58.497320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:58.554208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-786bedf8-bd2f-4b4c-936a-ea63324b23cd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:54:58.555410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-786bedf8-bd2f-4b4c-936a-ea63324b23cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:58.650775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 724/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:54:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:54:58.798934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:54:58.800427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:58.911146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:58.947893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c733b4-e743-4607-90c0-152e18cdc883 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:54:59.058127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:54:59.058656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:54:59.058927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:54:59.059800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:54:59.061394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:54:59.136405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:54:59.150460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:54:59.152058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bad40f-07be-480a-ad83-3e00bed9f6a8 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:54:59.169986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 701/1425] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:54:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:00.266438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:00.267006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:00.267006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:00.268373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 725/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:55:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:00.274905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a4b8-f8b4-491a-8dc2-1777821a82e3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:00.275303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a4b8-f8b4-491a-8dc2-1777821a82e3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:00.275459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a4b8-f8b4-491a-8dc2-1777821a82e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:00.351070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ae68a4b8-f8b4-491a-8dc2-1777821a82e3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:00.352274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ae68a4b8-f8b4-491a-8dc2-1777821a82e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:00.447957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 702/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:00.597248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:00.598827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:00.803249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:00.866879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a92c94e-c461-405c-b91d-f8157a401391 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:00.979027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:00.979644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:00.979918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:00.980186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:00.982409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:01.030096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:01.070721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:01.072316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c766799-8fec-4b42-8cd0-6f2ad1bd0f3f service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:01.089918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 726/1428] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:02.138619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:02.139209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:02.139209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:02.140494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 703/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:55:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:02.146662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbb907-7abe-4c6c-a5dc-47b45b951b7e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:02.147085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbb907-7abe-4c6c-a5dc-47b45b951b7e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:02.147218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbb907-7abe-4c6c-a5dc-47b45b951b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:02.223856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6bcbb907-7abe-4c6c-a5dc-47b45b951b7e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:02.225066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6bcbb907-7abe-4c6c-a5dc-47b45b951b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:02.321719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 727/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:55:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:02.470568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:02.472132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:02.520067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:02.616880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417b0df4-75a1-4a00-af06-4d9bbbce2ae3 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:02.726549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:02.727155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:02.727456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:02.727724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:02.729385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:02.736553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:02.819643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:02.821247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44988083-c0db-45d6-8ef2-e7200328b250 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:02.839379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 704/1431] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:03.891696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:03.892259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:03.892259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:03.893561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 728/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:03.899807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9d156-5ab3-4394-b5cd-542d69094e13 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:03.900259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9d156-5ab3-4394-b5cd-542d69094e13 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:03.900371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9d156-5ab3-4394-b5cd-542d69094e13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:03.976817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-efd9d156-5ab3-4394-b5cd-542d69094e13 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:03.978040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-efd9d156-5ab3-4394-b5cd-542d69094e13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:04.046515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:04.075522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 705/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:55:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:04.231775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:04.233766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:04.252956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:04.378942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc3d6ee-ac43-41ac-91c1-300cf9882610 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:04.466261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:04.492841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:04.493388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:04.493664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:04.493941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:04.495556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:04.582686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:04.584259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6982cf-a3ab-41c1-b7f2-c7e10f1ec718 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:04.602250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 729/1434] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:05.652554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:05.653116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:05.653116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:05.655259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 706/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:55:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:05.661436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1adf6-91f7-4a3c-b322-dd33f9b7c9b9 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:05.661889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1adf6-91f7-4a3c-b322-dd33f9b7c9b9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:05.662015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1adf6-91f7-4a3c-b322-dd33f9b7c9b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:05.737276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-12d1adf6-91f7-4a3c-b322-dd33f9b7c9b9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:05.738513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-12d1adf6-91f7-4a3c-b322-dd33f9b7c9b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:05.780686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:05.834919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 730/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:55:05.986295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:05.987744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:06.132048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacc2b56-b671-4296-ba6e-39844d97df26 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:06.186192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:06.244553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:06.245078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:06.245352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:06.245617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:06.247255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:06.337463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:06.339088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463fd56a-3219-4237-88a6-adb2f499d06d service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:06.357845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 707/1437] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:07.412303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:07.412818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:07.412818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:07.414191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 731/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:07.420728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1267e-97fc-4d8a-94ba-6e0ff52b3f38 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:07.421218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1267e-97fc-4d8a-94ba-6e0ff52b3f38 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:07.421628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1267e-97fc-4d8a-94ba-6e0ff52b3f38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:07.496539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-65b1267e-97fc-4d8a-94ba-6e0ff52b3f38 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:07.498274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-65b1267e-97fc-4d8a-94ba-6e0ff52b3f38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:07.500270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:07.593881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 708/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:55:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:07.743223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:07.744729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:07.888755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f034764-4976-4ce7-bc63-b75a73233c69 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:07.914780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:08.000181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:08.000816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:08.001095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:08.001367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:08.003012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:08.089490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:08.091107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf857e-9a17-46fd-8fcf-ef5e56b2d463 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:08.109288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 732/1440] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:10.163298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:10.163974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:10.163974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:10.165185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 709/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:10.170690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496da8bd-9b26-4bdd-879a-6b656e700141 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:10.171100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496da8bd-9b26-4bdd-879a-6b656e700141 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:10.171233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496da8bd-9b26-4bdd-879a-6b656e700141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:10.212170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:10.237595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-496da8bd-9b26-4bdd-879a-6b656e700141 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:10.238819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-496da8bd-9b26-4bdd-879a-6b656e700141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:10.334288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 733/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:55:10.620147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:10.621751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:10.723553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a4774-11a0-41ac-8e07-f428bb579230 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:10.755831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:10.796303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:10.796911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:10.797182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:10.797447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:10.799271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:10.909874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:10.911262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a366a7b-515a-41fa-92ff-094bc47a0429 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:10.929298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 710/1443] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:13.215120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:13.215729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:13.215729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:13.216994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 734/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:13.222452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1443d4b2-d74b-47da-a663-5eaca0ceb0e9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:13.222888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1443d4b2-d74b-47da-a663-5eaca0ceb0e9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:13.223005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1443d4b2-d74b-47da-a663-5eaca0ceb0e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:13.289208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1443d4b2-d74b-47da-a663-5eaca0ceb0e9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:13.290449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1443d4b2-d74b-47da-a663-5eaca0ceb0e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:13.303571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:13.388920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 711/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:13.675171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:13.676717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:13.778125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3246f10-8b78-48a1-95a1-751c3994a909 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:13.790562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:13.850787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:13.851393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:13.851671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:13.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:13.853533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:13.961482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:13.962858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd290d29-334a-4ed3-8655-0c9ba00b5231 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:13.981112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 735/1446] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:16.430362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:16.430930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:16.430930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:16.432233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 712/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:16.438180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e787b-e91e-4f9e-ad28-54d6ad235797 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:16.438608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e787b-e91e-4f9e-ad28-54d6ad235797 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:16.438718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6e787b-e91e-4f9e-ad28-54d6ad235797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:16.486873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:16.505738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-aa6e787b-e91e-4f9e-ad28-54d6ad235797 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:16.506961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-aa6e787b-e91e-4f9e-ad28-54d6ad235797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:16.603015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 736/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:16.888050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:16.889657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:16.992417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d3c6cb-a3bf-4577-9ccc-efa021089675 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:17.031901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:17.066556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:17.067168 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:17.067449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:17.067726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:17.069347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:17.175984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:17.177351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f6370-f63f-48bd-97f6-9a934da9e2e3 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:17.195589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 713/1449] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:19.425561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:19.426090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:19.426210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:19.427529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 737/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:19.432884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b49e9fc-547d-4fd5-8de2-ca751dab3baa None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:19.433300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b49e9fc-547d-4fd5-8de2-ca751dab3baa None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:19.433433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b49e9fc-547d-4fd5-8de2-ca751dab3baa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:19.500358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7b49e9fc-547d-4fd5-8de2-ca751dab3baa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:19.501614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7b49e9fc-547d-4fd5-8de2-ca751dab3baa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:19.522769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:19.600868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 714/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:19.888131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:19.889715 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:19.991689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1864ff58-c9a2-434e-882a-6fc46ef2866f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:20.012943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:20.067838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:20.068454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:20.068739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:20.069037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:20.070701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:20.178476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:20.179837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad03b51-5ad5-4e9e-8de1-e6c52363eda9 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:20.198584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 738/1452] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:22.461220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:22.461740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:22.461740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:22.463058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 715/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:22.468872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edde5618-8698-4c92-a2d9-990b7671b517 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:22.469322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edde5618-8698-4c92-a2d9-990b7671b517 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:22.469419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edde5618-8698-4c92-a2d9-990b7671b517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:22.531888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:22.534937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-edde5618-8698-4c92-a2d9-990b7671b517 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:22.536623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-edde5618-8698-4c92-a2d9-990b7671b517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:22.631072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 739/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:22.912187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:22.913730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:23.017948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef877e-03ab-4a6e-9fc3-2c5e1b99fc21 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:23.066982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:23.091118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:23.091749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:23.092031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:23.092298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:23.093904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:23.202008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:23.203557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb2328d-73f0-43f4-830f-186dddec19b2 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:23.221456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 716/1455] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:25.415342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:25.415814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:25.415937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:25.419890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 740/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:25.422932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad93b9-5e43-4e48-abb3-f1b1dc74c060 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:25.423340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad93b9-5e43-4e48-abb3-f1b1dc74c060 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:25.423447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad93b9-5e43-4e48-abb3-f1b1dc74c060 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:25.488092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4aad93b9-5e43-4e48-abb3-f1b1dc74c060 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:25.489275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4aad93b9-5e43-4e48-abb3-f1b1dc74c060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:25.519119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:25.586060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 717/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:25.874617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:25.876157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:25.977590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5134b81-b03e-45aa-acea-06cc15cd18d8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:26.006583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:26.051460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:26.052065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:26.052343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:26.052613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:26.054252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:26.161738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:26.163134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9a7f70-d2b8-48c2-b74e-dbb4cccb67bc service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:26.181337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 741/1458] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:28.370360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:28.370921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:28.370921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:28.372281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 718/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:28.377706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3518971-f093-47f6-b53d-d4c4e33c1651 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:28.378139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3518971-f093-47f6-b53d-d4c4e33c1651 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:28.378306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3518971-f093-47f6-b53d-d4c4e33c1651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:28.443492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f3518971-f093-47f6-b53d-d4c4e33c1651 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:28.444675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f3518971-f093-47f6-b53d-d4c4e33c1651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:28.452804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:28.539837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 742/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:28.823734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:28.825917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:28.928174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45585a0-aa6a-40e8-acb7-3d7fe6a4d554 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:28.985361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:29.002975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:29.003577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:29.003854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:29.004127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:29.005722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:29.119003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:29.120734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1cb34d-f8e4-4e3d-bd25-77ff7d135f04 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:29.139462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 719/1461] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:31.339947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:31.340477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:31.340477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:31.342555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 743/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:31.347900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f3c66-984f-418c-b5c2-cb396107cb0d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:31.348300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f3c66-984f-418c-b5c2-cb396107cb0d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:31.348443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f3c66-984f-418c-b5c2-cb396107cb0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:31.414092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f78f3c66-984f-418c-b5c2-cb396107cb0d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:31.415291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f78f3c66-984f-418c-b5c2-cb396107cb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:31.452927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:31.510471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 720/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:31.793084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:31.794677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:31.896772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc79d48c-d823-4a97-8919-99c426ad01a2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:31.933826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:31.970785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:31.971386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:31.971663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:31.971927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:31.973491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:32.081049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:32.082520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447e89b7-c0e6-449f-beba-331283de8321 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:32.100508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 744/1464] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:34.262159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:34.263215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:34.263336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:34.264588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 721/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:34.269866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7dd40-d10b-4d05-b3a7-cf21f45578f1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:34.270320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7dd40-d10b-4d05-b3a7-cf21f45578f1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:34.270442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7dd40-d10b-4d05-b3a7-cf21f45578f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:34.334967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9ba7dd40-d10b-4d05-b3a7-cf21f45578f1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:34.336168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9ba7dd40-d10b-4d05-b3a7-cf21f45578f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:34.354214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:34.431403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 745/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:34.715245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:34.717295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:34.821445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb64a6-fc8e-4457-aeb4-90b53d323e4c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:34.889839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:34.896787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:34.897391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:34.897667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:34.897938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:34.899555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:35.007035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:35.008403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c1aab7-7870-4352-9468-9ae438ac095c service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:35.026733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 722/1467] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:37.170173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:37.170749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:37.170749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:37.172052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 746/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:55:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:37.177403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc55da6-e142-49e4-b588-7f6c0f256756 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:37.177861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc55da6-e142-49e4-b588-7f6c0f256756 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:37.177962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc55da6-e142-49e4-b588-7f6c0f256756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:37.242986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ebc55da6-e142-49e4-b588-7f6c0f256756 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:37.244184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ebc55da6-e142-49e4-b588-7f6c0f256756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:37.293337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:37.341758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 723/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:37.621253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:37.623324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:37.723915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff43deb-c4c1-4967-975b-8106c326ae5c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:37.771697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:37.797176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:37.797774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:37.798074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:37.798347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:37.799917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:37.907065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:37.908429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9806e6-e69e-43ac-9043-1674936492aa service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:37.926499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 747/1470] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:47.339635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=22 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:47.924277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=66 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:51.686393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394df6-6c4a-4a70-9bd2-0fbfb169b6f5 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:51.686969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394df6-6c4a-4a70-9bd2-0fbfb169b6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:51.686969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa394df6-6c4a-4a70-9bd2-0fbfb169b6f5 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:51.688294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 724/1471] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:55:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:51.693552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f068c8-ff24-4ad8-b1e8-65990690d389 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:51.694008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f068c8-ff24-4ad8-b1e8-65990690d389 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:51.694222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f068c8-ff24-4ad8-b1e8-65990690d389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:51.759664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-41f068c8-ff24-4ad8-b1e8-65990690d389 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:51.760857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-41f068c8-ff24-4ad8-b1e8-65990690d389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:51.855354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 748/1472] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:52.146121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:52.146404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:52.146703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:52.148076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 725/1473] 10.222.0.46 () {54 vars in 962 bytes} [Fri Oct 28 10:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:52.153158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba4c483-eccf-4614-8caa-452555afc05d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:52.153444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba4c483-eccf-4614-8caa-452555afc05d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:52.153722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba4c483-eccf-4614-8caa-452555afc05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:52.229642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3ba4c483-eccf-4614-8caa-452555afc05d None None] MFA Rules not processed for user `589508bdaf3c4e159c01e3a43eb69392`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:52.231489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3ba4c483-eccf-4614-8caa-452555afc05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:52.252721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:52.306925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 749/1474] 10.222.0.46 () {58 vars in 1059 bytes} [Fri Oct 28 10:55:52 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:52.315201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:52.316738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:52.418744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7434d82a-122c-4ebe-b633-fa65e1672efe None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:52.494196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:52.494797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:52.495072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:52.495341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:52.497002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:52.550458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:52.604469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:52.606710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766d0858-4cee-4b65-aa66-8d8cb4b93562 service glance] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:52.624926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 726/1475] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:55:54.814424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:54.814987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:54.814987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:54.816307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 750/1476] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:55:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:54.821542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b24ab6-0ffc-49b4-9aad-6235bf1d425d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:54.822063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b24ab6-0ffc-49b4-9aad-6235bf1d425d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:54.822162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b24ab6-0ffc-49b4-9aad-6235bf1d425d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:54.888240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-07b24ab6-0ffc-49b4-9aad-6235bf1d425d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:54.889445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-07b24ab6-0ffc-49b4-9aad-6235bf1d425d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:54.984525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 727/1477] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:55:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:55.250423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:55.251951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:55.374736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4139b6ed-26d6-40d0-bac3-67d62089f578 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:55.423507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:55.467305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:55.467908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:55.468186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:55.468456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:55.469104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:55.470867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:55:55.470867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 10:55:55.470867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 10:55:55.472000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc52847d-4df3-4182-90f0-089e2c93162a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:55.490775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 751/1478] 10.222.0.46 () {56 vars in 1108 bytes} [Fri Oct 28 10:55:55 2022] GET /identity/v3/projects => generated 2553 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 10:55:57.629997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:57.630558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:57.630558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:57.631923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 728/1479] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:55:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:55:57.637175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7b16e-8805-417b-a571-acbfcc521b43 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:57.637608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7b16e-8805-417b-a571-acbfcc521b43 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:57.637712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7b16e-8805-417b-a571-acbfcc521b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:57.703167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0fe7b16e-8805-417b-a571-acbfcc521b43 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:55:57.704388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0fe7b16e-8805-417b-a571-acbfcc521b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:57.800286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 752/1480] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:55:58.090253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:55:58.091811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:58.101622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:58.237621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73d054e-13b5-4ca9-9e34-f828de4d0556 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:55:58.316780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:55:58.350657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:55:58.351192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:55:58.351466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:55:58.351731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:55:58.353303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:55:58.441786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:55:58.443436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553fd8a4-7ceb-474b-a748-0a4818fbe817 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:55:58.461571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 729/1481] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:55:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:56:00.748631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:00.749207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:00.749207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:00.750557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 753/1482] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:56:00.755672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ad8ac-a799-4de0-8b94-0356f576d39c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:00.756075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ad8ac-a799-4de0-8b94-0356f576d39c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:00.756221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ad8ac-a799-4de0-8b94-0356f576d39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:00.822141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d44ad8ac-a799-4de0-8b94-0356f576d39c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:56:00.823358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d44ad8ac-a799-4de0-8b94-0356f576d39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:00.871926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:00.921493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 730/1483] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:56:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:56:01.206708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:56:01.208280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:01.232427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:01.311367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c50596-8def-4d7b-a715-b0abebe2becc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:56:01.383747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:56:01.384347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:01.384621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:01.384886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:01.387022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:01.458586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:01.492987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:56:01.494445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e84098f-d632-475f-995c-d0b97b3406fe service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:56:01.513407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 754/1484] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:56:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:56:03.686364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:03.686901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:03.686901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:03.688275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 731/1485] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:56:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:56:03.693664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa782823-4725-4f87-abb3-68450c9c5bef None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:03.694151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa782823-4725-4f87-abb3-68450c9c5bef None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:03.694268 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa782823-4725-4f87-abb3-68450c9c5bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:03.760058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-fa782823-4725-4f87-abb3-68450c9c5bef None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:56:03.761285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fa782823-4725-4f87-abb3-68450c9c5bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:03.856462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 755/1486] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:56:04.141041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:56:04.142626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:04.247028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbbf375-6910-4606-a6fd-5517162e806a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:56:04.320411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:04.322156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:56:04.322888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:04.323358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:04.324364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:04.326182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:04.433930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:56:04.435329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a8345-1a85-4564-9ec8-49cb343afa02 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:56:04.453634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 732/1487] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:56:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:56:06.733085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:06.733988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:06.733988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:06.735381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 756/1488] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:56:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:56:06.740754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261fd308-c92e-454b-9d95-55e580dbc1b3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:06.741194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261fd308-c92e-454b-9d95-55e580dbc1b3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:06.741299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261fd308-c92e-454b-9d95-55e580dbc1b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:06.806495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-261fd308-c92e-454b-9d95-55e580dbc1b3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:56:06.807760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-261fd308-c92e-454b-9d95-55e580dbc1b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:06.859654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:06.906147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 733/1489] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:56:07.219545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:56:07.221085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:07.260596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:07.333527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5582c78-b27f-41e9-92e1-9074d7290569 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:56:07.409195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:56:07.409806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:07.410115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:07.410389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:07.412451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:07.494113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:07.523074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:56:07.524523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5a619-67e1-4dfb-83dc-7e7f0dbd1a3f service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:56:07.543432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 757/1490] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:56:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:56:09.875223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:09.875787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:09.875787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:09.877122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 734/1491] 10.222.0.46 () {54 vars in 927 bytes} [Fri Oct 28 10:56:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:56:09.882372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf9af82-4b6f-4dee-a3a8-1699db7d1386 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:09.882828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf9af82-4b6f-4dee-a3a8-1699db7d1386 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:09.882943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf9af82-4b6f-4dee-a3a8-1699db7d1386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:09.947452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0cf9af82-4b6f-4dee-a3a8-1699db7d1386 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:56:09.949383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0cf9af82-4b6f-4dee-a3a8-1699db7d1386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:10.046025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 758/1492] 10.222.0.46 () {58 vars in 1024 bytes} [Fri Oct 28 10:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:56:10.330875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:56:10.332432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:10.435082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19da077-9763-43df-bd0e-dc5a8bef149a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:56:10.510207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:56:10.511421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:56:10.511707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:56:10.511982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:56:10.513595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:56:10.522066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:10.622337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:56:10.623789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba01cd95-6d53-48ae-aecc-6e670a709541 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:56:10.641854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 735/1493] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:56:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:56:20.044486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=82 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:56:20.639665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=50 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:52.323183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:52.323683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:52.323788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:52.399151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:57:52.400354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:52.495399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 759/1494] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 10:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:57:52.503267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:52.503744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:52.503862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:52.577649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:57:52.578869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:52.674982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 736/1495] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 10:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:57:52.679537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:52.680732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:52.692051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ade8c-6252-439f-b069-1d964ac247ad None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:52.753681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:52.787182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:52.787793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:52.788076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:52.788347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:52.789563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 10:57:52.789867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:52.790907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:52.790907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 10:57:52.790907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 10:57:52.791794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:52.803042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 760/1496] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 10:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:57:52.813026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:52.822272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047035af-70eb-4a87-97e3-01532c95a1be None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:52.883682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:52.918057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:52.918665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:52.918949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:52.919218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:52.919840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:52.921641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:52.921641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 10:57:52.921641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 10:57:52.922897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:52.982479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 737/1497] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 10:57:52 2022] POST /identity/v3/projects => generated 379 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 10:57:52.986846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:52.988425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.021623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f122054-dffb-4aa0-bbdf-46d422da5149 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:53.115378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:53.115985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.116263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.116532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.117766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:53.118802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:53.118802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 10:57:53.118802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 10:57:53.120010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:53.161661 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 761/1498] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 10:57:52 2022] POST /identity/v3/users => generated 290 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 10:57:53.166349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:53.167933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.230221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e255ad-2a1e-4a56-ba0e-566ba0e1f67d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:53.314881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:53.326874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:53.327483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.327761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.328040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.328684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:53.329730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:53.329730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 10:57:53.329730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 10:57:53.330654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:53.342761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 738/1499] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 10:57:53 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 10:57:53.346784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:53.348299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.359852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9452c4-18af-48b5-baa4-8e68e2a4c7b2 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:53.385908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:53.454222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:53.454829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.455104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.455369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.455995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:53.456986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:53.456986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 10:57:53.456986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 10:57:53.457863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c28ac1c-16a3-455c-87f2-5b58f03c92cf admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:53.469893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 762/1500] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 10:57:53 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 10:57:53.474000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:53.475537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.509480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11c56-6291-4cac-9bed-a8157c330fcd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:53.602053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:53.602712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.602985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.603277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] PATH_INFO: `/v3/projects/2bece738ecf34133ba45bfee258d654f/users/af9a48a5ddb84813bb75bc86afaeb4d3/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.625820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bece738ecf34133ba45bfee258d654f, user_id=af9a48a5ddb84813bb75bc86afaeb4d3, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:53.626885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:53.626885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 10:57:53.626885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 10:57:53.628806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05312b-43b0-404f-ac18-2fec5171af1a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:53.653587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 739/1501] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 10:57:53 2022] PUT /identity/v3/projects/2bece738ecf34133ba45bfee258d654f/users/af9a48a5ddb84813bb75bc86afaeb4d3/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:57:53.665021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ed328-e38c-40e9-b366-71fa137bf038 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.665315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ed328-e38c-40e9-b366-71fa137bf038 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.665616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ed328-e38c-40e9-b366-71fa137bf038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.721518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cf2ed328-e38c-40e9-b366-71fa137bf038 None None] MFA Rules not processed for user `af9a48a5ddb84813bb75bc86afaeb4d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:57:53.722743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cf2ed328-e38c-40e9-b366-71fa137bf038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.797234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 763/1502] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 10:57:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:57:53.810289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.810576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.810851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.812173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 740/1503] 10.222.0.46 () {54 vars in 957 bytes} [Fri Oct 28 10:57:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:57:53.816950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54453bcb-9ef9-41e9-8cc7-dca83fb62837 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:53.817268 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54453bcb-9ef9-41e9-8cc7-dca83fb62837 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:53.817543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54453bcb-9ef9-41e9-8cc7-dca83fb62837 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:53.850195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:53.894032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-54453bcb-9ef9-41e9-8cc7-dca83fb62837 None None] MFA Rules not processed for user `2297b30fe14d4de8a3f39e247ed59dfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:57:53.895185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-54453bcb-9ef9-41e9-8cc7-dca83fb62837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:53.969579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 764/1504] 10.222.0.46 () {58 vars in 1054 bytes} [Fri Oct 28 10:57:53 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:57:53.977231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:53.978833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:54.028563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:54.082250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87751e3-377b-4e0d-88b0-dea20d794c2e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:54.154541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:54.155104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:54.155380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:54.155651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:54.157242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:54.372484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:54.373910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:54.377376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:54.392703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 741/1505] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:57:53 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:57:54.438186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235f05e-5f4c-48b0-bbbf-db7b5b0b5f74 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:54.438609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235f05e-5f4c-48b0-bbbf-db7b5b0b5f74 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:54.438736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235f05e-5f4c-48b0-bbbf-db7b5b0b5f74 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:54.439995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 765/1506] 10.222.0.46 () {52 vars in 830 bytes} [Fri Oct 28 10:57:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:57:54.443461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:54.444686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:54.542137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d69873f-5140-4b09-ab3b-2ffde54c0e24 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:54.633247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:54.633867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:54.634167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:54.634437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] PATH_INFO: `/v3/users/af9a48a5ddb84813bb75bc86afaeb4d3` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:54.646835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af9a48a5ddb84813bb75bc86afaeb4d3)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:54.647859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:54.647859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 10:57:54.647859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 10:57:54.649059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:54.658377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:54.713520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af9a48a5ddb84813bb75bc86afaeb4d3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:54.713995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af9a48a5ddb84813bb75bc86afaeb4d3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:54.723764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af9a48a5ddb84813bb75bc86afaeb4d3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:54.724215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af9a48a5ddb84813bb75bc86afaeb4d3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:54.739115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af9a48a5ddb84813bb75bc86afaeb4d3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:54.766640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 742/1507] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 10:57:54 2022] DELETE /identity/v3/users/af9a48a5ddb84813bb75bc86afaeb4d3 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:57:54.770795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6133d1-e319-4016-b729-2356a89786be None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:54.771081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6133d1-e319-4016-b729-2356a89786be None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:54.771356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6133d1-e319-4016-b729-2356a89786be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:54.830276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-de6133d1-e319-4016-b729-2356a89786be None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:57:54.831461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-de6133d1-e319-4016-b729-2356a89786be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:54.907785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:54.921182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 766/1508] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 10:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:57:54.931159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:54.932390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:55.033224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3637c76-907f-4c62-9835-64d2a5e88fa9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:55.087062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:57:55.108859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:55.109462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:55.109731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:55.110021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:55.111634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:55.217392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:55.218835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3b7984-46df-4d53-be9a-665d1b5d4580 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:55.237343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 743/1509] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:57:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:57:55.562604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6133d1-e319-4016-b729-2356a89786be None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:57:55.563869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-de6133d1-e319-4016-b729-2356a89786be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:57:55.576429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6133d1-e319-4016-b729-2356a89786be None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:57:55.670614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:57:55.671253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:57:55.671531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:57:55.671805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] PATH_INFO: `/v3/projects/2bece738ecf34133ba45bfee258d654f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:57:55.681997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bece738ecf34133ba45bfee258d654f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:57:55.683009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 10:57:55.683009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 10:57:55.683009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 10:57:55.684219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:57:55.740440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bece738ecf34133ba45bfee258d654f', 'request_id': 'req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:55.748384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bece738ecf34133ba45bfee258d654f', 'request_id': 'req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:55.761684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bece738ecf34133ba45bfee258d654f', 'request_id': 'req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:55.787422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] The token cache is being invalidate because project 2bece738ecf34133ba45bfee258d654f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 10:57:55.787712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:55.788168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f92ec45a-ca0c-4495-b3bc-8e3fc011e22d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 10:57:55.789548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 767/1510] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 10:57:55 2022] DELETE /identity/v3/projects/2bece738ecf34133ba45bfee258d654f => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 10:58:05.235188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=62 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:58:05.785458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:59:20.584463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:20.585065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:20.585065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:20.586359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 744/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:59:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 10:59:20.591242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fcddc-39bc-4cdc-a37e-7a8c11abbedf None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:20.591706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fcddc-39bc-4cdc-a37e-7a8c11abbedf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:20.591838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fcddc-39bc-4cdc-a37e-7a8c11abbedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:20.670621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3c0fcddc-39bc-4cdc-a37e-7a8c11abbedf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:20.671819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3c0fcddc-39bc-4cdc-a37e-7a8c11abbedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:20.774141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 768/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:59:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:59:20.934717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:20.936856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:21.053928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6293994-dd93-4494-a234-3e76a21055bb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:21.059444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:21.060050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:21.060327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:21.060601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:21.069050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:21.070686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7727948a-1eb3-4517-93a0-f8d090c36907 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:21.079901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 745/1513] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:22.155317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f07d97-3244-43cc-adc6-0425913573f1 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:22.155781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f07d97-3244-43cc-adc6-0425913573f1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:22.155781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f07d97-3244-43cc-adc6-0425913573f1 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:22.158251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 769/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:59:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:22.165148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afd859-488c-410f-8d25-c46f0ea2379e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:22.165661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afd859-488c-410f-8d25-c46f0ea2379e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:22.165791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23afd859-488c-410f-8d25-c46f0ea2379e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:22.187090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-23afd859-488c-410f-8d25-c46f0ea2379e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:22.188351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-23afd859-488c-410f-8d25-c46f0ea2379e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:22.196322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 746/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:59:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:59:22.344059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f07d97-3244-43cc-adc6-0425913573f1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:22.359428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f07d97-3244-43cc-adc6-0425913573f1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:22.365616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:22.366259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:22.366541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:22.366813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:22.374898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:22.376469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a023d957-fed3-4f35-809f-95e2e2d71586 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:22.385630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 770/1516] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:22.759968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9d3e45-c17b-4215-82bf-16edd8a87a78 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:22.760185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9d3e45-c17b-4215-82bf-16edd8a87a78 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:22.760429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9d3e45-c17b-4215-82bf-16edd8a87a78 None None] PATH_INFO: `/v3` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:22.761734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 747/1517] 10.222.0.25 () {54 vars in 925 bytes} [Fri Oct 28 10:59:22 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 10:59:22.768085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d4eee-c45a-4ab3-9bd4-d85c18f550a5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:22.768368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d4eee-c45a-4ab3-9bd4-d85c18f550a5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:22.768641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d4eee-c45a-4ab3-9bd4-d85c18f550a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:22.827944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e70d4eee-c45a-4ab3-9bd4-d85c18f550a5 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:22.829133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e70d4eee-c45a-4ab3-9bd4-d85c18f550a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:22.845992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:59:22.901378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 771/1518] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Oct 28 10:59:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 10:59:26.473823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab8fbd1-ecf2-4c16-aa5e-afba939c2031 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:26.474402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab8fbd1-ecf2-4c16-aa5e-afba939c2031 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:26.474402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab8fbd1-ecf2-4c16-aa5e-afba939c2031 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:26.475714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 748/1519] 10.222.0.46 () {54 vars in 961 bytes} [Fri Oct 28 10:59:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:26.480881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb8847-eaa5-47d9-b871-2626effb893b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:26.481295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb8847-eaa5-47d9-b871-2626effb893b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:26.481593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeb8847-eaa5-47d9-b871-2626effb893b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:26.501690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-bdeb8847-eaa5-47d9-b871-2626effb893b None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:26.502931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bdeb8847-eaa5-47d9-b871-2626effb893b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:26.512143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 772/1520] 10.222.0.46 () {58 vars in 1058 bytes} [Fri Oct 28 10:59:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:26.519959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab8fbd1-ecf2-4c16-aa5e-afba939c2031 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:26.533589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab8fbd1-ecf2-4c16-aa5e-afba939c2031 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:26.539206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:26.539805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:26.540078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:26.540346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:26.548747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:26.550344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e901026-d9c7-4392-b930-656c1c630ec1 service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:26.560341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 749/1521] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 10:59:33.843133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d2bcb-3f68-4f74-97d4-811dab18119c None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:33.843678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d2bcb-3f68-4f74-97d4-811dab18119c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:33.843678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d2bcb-3f68-4f74-97d4-811dab18119c None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:33.845025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 773/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:33.851724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fb7c7-177b-403f-b68d-211383f08ad1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:33.852148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fb7c7-177b-403f-b68d-211383f08ad1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:33.852254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fb7c7-177b-403f-b68d-211383f08ad1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:33.872276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ae3fb7c7-177b-403f-b68d-211383f08ad1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:33.873456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ae3fb7c7-177b-403f-b68d-211383f08ad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:33.881137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 750/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:59:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:34.034299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13d2bcb-3f68-4f74-97d4-811dab18119c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:34.049073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13d2bcb-3f68-4f74-97d4-811dab18119c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:34.054358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:34.054952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:34.055229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:34.055497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:34.063287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:34.064852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03bc24a-9b63-4279-b993-2d15ffef5125 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:34.074731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 774/1524] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:35.131751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd854eea-a989-4a50-b9f2-0dff2537ca48 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:35.132296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd854eea-a989-4a50-b9f2-0dff2537ca48 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:35.132296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd854eea-a989-4a50-b9f2-0dff2537ca48 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:35.133614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 751/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 10:59:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:35.139611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b62c62-db1f-4b66-93eb-b559de79b9b3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:35.140068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b62c62-db1f-4b66-93eb-b559de79b9b3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:35.140172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b62c62-db1f-4b66-93eb-b559de79b9b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:35.160615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e2b62c62-db1f-4b66-93eb-b559de79b9b3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:35.162405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e2b62c62-db1f-4b66-93eb-b559de79b9b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:35.170356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 775/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 10:59:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:35.319152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd854eea-a989-4a50-b9f2-0dff2537ca48 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:35.333016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd854eea-a989-4a50-b9f2-0dff2537ca48 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:35.338816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:35.339401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:35.339675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:35.339942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:35.347785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:35.349964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964d0481-bf85-4750-929b-e9ac73507cba service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:35.359532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 752/1527] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:39.128282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:39.130170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:39.130170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:39.130170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 776/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:59:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:39.135477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1793422-d056-476b-8c58-d6c85625426a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:39.135943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1793422-d056-476b-8c58-d6c85625426a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:39.136043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1793422-d056-476b-8c58-d6c85625426a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:39.155572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f1793422-d056-476b-8c58-d6c85625426a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:39.156763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f1793422-d056-476b-8c58-d6c85625426a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:39.165400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 753/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:59:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:39.449519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:39.451247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:39.529183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d765a-1cd6-48f2-b7b6-aac366cf4748 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:39.532804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:39.533409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:39.533688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:39.533961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:39.543229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:39.544683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57243be-9012-4e32-8f0f-2b1fbea7a88f service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:39.554054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 777/1530] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:41.673176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:41.673778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:41.673778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:41.675086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 754/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:59:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:41.680226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee628c66-e14c-47aa-96f7-5bc795e30ab3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:41.680659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee628c66-e14c-47aa-96f7-5bc795e30ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:41.680768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee628c66-e14c-47aa-96f7-5bc795e30ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:41.700338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ee628c66-e14c-47aa-96f7-5bc795e30ab3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:41.701528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ee628c66-e14c-47aa-96f7-5bc795e30ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:41.709848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 778/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:59:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:41.991047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:41.992723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:42.005953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79d5460-482d-4000-bd1f-03bea4c8bfbf None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:42.009422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:42.010043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:42.010322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:42.010590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:42.019249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:42.020694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514ad90f-5486-4eda-9986-700ed14ae44e service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:42.029838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 755/1533] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:45.662455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:45.663045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:45.663045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:45.664323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 779/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:59:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:45.669412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e72dd-4cd4-49c2-a8d8-e61d7a449731 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:45.669954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e72dd-4cd4-49c2-a8d8-e61d7a449731 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:45.669954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e72dd-4cd4-49c2-a8d8-e61d7a449731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:45.678019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=81 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:59:45.690515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-590e72dd-4cd4-49c2-a8d8-e61d7a449731 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:45.691700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-590e72dd-4cd4-49c2-a8d8-e61d7a449731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:45.700067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 756/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:45.993600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:45.995297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:46.059374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:59:46.073016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089bb426-fb38-4720-9192-dc52a3fc1f29 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:46.076660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:46.077270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:46.077553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:46.077830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:46.086761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:46.088199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8660baa-6f88-460f-a71e-cfbd54d1efdf service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:46.097745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 780/1536] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:48.232402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:48.233257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:48.233257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:48.234580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 757/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 10:59:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 10:59:48.239689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e9d5-233b-4afc-a6ea-68c84b1414aa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:48.240138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e9d5-233b-4afc-a6ea-68c84b1414aa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:48.240248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e9d5-233b-4afc-a6ea-68c84b1414aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:48.260145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a7c3e9d5-233b-4afc-a6ea-68c84b1414aa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 10:59:48.261344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a7c3e9d5-233b-4afc-a6ea-68c84b1414aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:48.269221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 781/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 10:59:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 10:59:48.567138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 10:59:48.568803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 10:59:48.581510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee19a1-2fab-46c0-93e1-db80e21d36f9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 10:59:48.584970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 10:59:48.585560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 10:59:48.585836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 10:59:48.586141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 10:59:48.594803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 10:59:48.596219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c5abf-5185-454d-855b-f3892ccbb158 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 10:59:48.605866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 758/1539] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 10:59:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 10:59:58.255150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=11 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 10:59:58.601770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=8 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:11.822642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:11.823275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:11.823275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:11.847483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:11.849239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:11.861292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 782/1540] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:11.870557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:11.871563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:11.872055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:11.898552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:11.900117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:11.917504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 759/1541] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:11.922869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:11.939503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521c7652-6fd7-4b14-9fa7-34f0d702163e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:11.949657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:11.950325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:11.950606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:11.950881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:11.951536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:11.951839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:11.953446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:11.972373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 783/1542] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:11.982598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:11.998732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89c9720-5465-4564-841d-83b9b4f79baf None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.004848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.005891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.006571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.007081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.008012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.010373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837e84f5-c1f5-4063-bc80-0f66cd984bb0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.086087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 760/1543] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:11 2022] POST /identity/v3/projects => generated 377 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:12.091476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.096623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b885a4-bf92-40d6-a62a-9d8bdf49fd6b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.096960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b885a4-bf92-40d6-a62a-9d8bdf49fd6b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.097179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b885a4-bf92-40d6-a62a-9d8bdf49fd6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.112360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d889c2-4212-4743-8c96-12a67bd27f6e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.141324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-00b885a4-bf92-40d6-a62a-9d8bdf49fd6b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.142574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-00b885a4-bf92-40d6-a62a-9d8bdf49fd6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.242053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.242693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.242977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.243246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.243897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.253767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.307848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 784/1544] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/users => generated 289 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:12.310688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.315419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 761/1545] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 28 11:01:12.333856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07616186-d60e-4503-bb86-7138f87b556b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.334255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07616186-d60e-4503-bb86-7138f87b556b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.334383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5c6756-d24b-4f4f-8761-c117c8b857f9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.334473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07616186-d60e-4503-bb86-7138f87b556b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.347710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.348848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.349140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.349414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.350351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.351970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.372166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-07616186-d60e-4503-bb86-7138f87b556b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.372596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 785/1546] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:12.373439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-07616186-d60e-4503-bb86-7138f87b556b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.377322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.392398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 762/1547] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.394351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967c32b2-00b4-4ca2-b54a-ed72a41e62e6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.403343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.403978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.404733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.405018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.405671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.407339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.411559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07616186-d60e-4503-bb86-7138f87b556b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.421376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07616186-d60e-4503-bb86-7138f87b556b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.424185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 786/1548] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:12.430074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.443876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.444167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.444267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.444493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.445114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:12.445416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.446758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:12.446758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 11:01:12.446758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 11:01:12.447041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232c659-3f54-4303-8a0c-ad60a42f27c9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.447708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5babb562-5bd4-48e2-a5a7-ef016800ff22 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.457510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.459420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.459707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.459984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] PATH_INFO: `/v3/projects/33be387e591543919673f57dc809f643/users/49e38ef4721d4011a2a82f84b0dcdb78/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.465269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 763/1549] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:01:12.480873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92cc0c-1080-4bc7-83b8-414e1e0e359d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.481237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92cc0c-1080-4bc7-83b8-414e1e0e359d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.481602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92cc0c-1080-4bc7-83b8-414e1e0e359d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.492101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33be387e591543919673f57dc809f643, user_id=49e38ef4721d4011a2a82f84b0dcdb78, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.493177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:12.493177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 11:01:12.493177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 11:01:12.495161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.516014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7c92cc0c-1080-4bc7-83b8-414e1e0e359d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.516827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7c92cc0c-1080-4bc7-83b8-414e1e0e359d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.526313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 787/1550] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:12 2022] PUT /identity/v3/projects/33be387e591543919673f57dc809f643/users/49e38ef4721d4011a2a82f84b0dcdb78/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:12.527153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.600386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 764/1551] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.600386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f1f57-0276-4439-8ec1-fc1183422c6e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.600386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f1f57-0276-4439-8ec1-fc1183422c6e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.600386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f1f57-0276-4439-8ec1-fc1183422c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.609898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cc121e-ead9-4643-9e48-8e84cd3c23d5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.615067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.615670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.615946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.616215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.616842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.617856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:12.617856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 11:01:12.617856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 11:01:12.618494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b66f1f57-0276-4439-8ec1-fc1183422c6e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.619217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad8a2c9-0d61-4c23-9140-a50a7d425509 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.619689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b66f1f57-0276-4439-8ec1-fc1183422c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.627803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 765/1552] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.631548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d548c-354d-4ca3-880b-4aab433b08ba None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.631825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d548c-354d-4ca3-880b-4aab433b08ba None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.632092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d548c-354d-4ca3-880b-4aab433b08ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.651119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-470d548c-354d-4ca3-880b-4aab433b08ba None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.652308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-470d548c-354d-4ca3-880b-4aab433b08ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.658018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 788/1553] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/projects => generated 377 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:12.658018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c79b88-2592-436c-899b-f5407abd1389 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.658018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c79b88-2592-436c-899b-f5407abd1389 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.658018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c79b88-2592-436c-899b-f5407abd1389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.677529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-30c79b88-2592-436c-899b-f5407abd1389 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.678830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-30c79b88-2592-436c-899b-f5407abd1389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.683367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:12.687298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:12.714738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 766/1554] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.716231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47888221-01d6-448a-85ce-b9722929cd28 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.716690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47888221-01d6-448a-85ce-b9722929cd28 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.717122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47888221-01d6-448a-85ce-b9722929cd28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.736184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-47888221-01d6-448a-85ce-b9722929cd28 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.737380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-47888221-01d6-448a-85ce-b9722929cd28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.743120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 789/1555] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.744370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd095cf4-4903-4865-9b60-16c6daf13477 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.744656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd095cf4-4903-4865-9b60-16c6daf13477 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.744929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd095cf4-4903-4865-9b60-16c6daf13477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.752595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 767/1556] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.753815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911dba-0d41-43ee-94de-8caacfb51f4d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.754115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911dba-0d41-43ee-94de-8caacfb51f4d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.754384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78911dba-0d41-43ee-94de-8caacfb51f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.773607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-78911dba-0d41-43ee-94de-8caacfb51f4d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.774824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-78911dba-0d41-43ee-94de-8caacfb51f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.780060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cd095cf4-4903-4865-9b60-16c6daf13477 None None] MFA Rules not processed for user `49e38ef4721d4011a2a82f84b0dcdb78`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.781241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cd095cf4-4903-4865-9b60-16c6daf13477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.782618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 768/1557] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.783917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f014d1-7db0-4eaa-9578-997a7f0d8621 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.784278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f014d1-7db0-4eaa-9578-997a7f0d8621 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.784709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f014d1-7db0-4eaa-9578-997a7f0d8621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.804361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-22f014d1-7db0-4eaa-9578-997a7f0d8621 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.805540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-22f014d1-7db0-4eaa-9578-997a7f0d8621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.813506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 769/1558] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.814778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.815100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.815373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.834807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.835995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.843858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 770/1559] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.844400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.850790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 790/1560] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.852029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db750e-7404-4da8-8279-a804f1c236c0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.852315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db750e-7404-4da8-8279-a804f1c236c0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.852586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db750e-7404-4da8-8279-a804f1c236c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.856650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e660b25e-3917-4001-8a5e-2fc0cc560ccb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.861394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.862017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.862286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.862555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.863212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:12.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 11:01:12.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 11:01:12.866072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1cad90-9c34-4fb7-975a-45a98a32c45d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.872818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-86db750e-7404-4da8-8279-a804f1c236c0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.874021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-86db750e-7404-4da8-8279-a804f1c236c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.883780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 791/1561] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.885184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a4eef5-c2b6-4d65-9c12-799b6bd96a4a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.885537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a4eef5-c2b6-4d65-9c12-799b6bd96a4a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.886037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a4eef5-c2b6-4d65-9c12-799b6bd96a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.890626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 771/1562] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/users => generated 289 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:12.891746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.892050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.892311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.910784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b7a4eef5-c2b6-4d65-9c12-799b6bd96a4a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.911508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.912044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b7a4eef5-c2b6-4d65-9c12-799b6bd96a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.912683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.920187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 792/1563] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.920538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 772/1564] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.921335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.921439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.921652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.921924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.933801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e98d8-6b97-4a87-be03-9cef449dd6e0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.938650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.939260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.939539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.939813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.940471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:12.940770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.943064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.945301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:12.947108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:12.954881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 773/1565] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:12.958868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 793/1566] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:12.959003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.959086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:12.970176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c71a49-afaa-4010-b3e3-07173adadca2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.970418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d733d98a-3490-439a-9cbd-e66ce095d182 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:12.976217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.976391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:12.976875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.977043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:12.977211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.977350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:12.977443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.977658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:12.978033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.978511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:12.978816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:12.980059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.980387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:12.993661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 774/1567] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:12.994513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.005344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a9fdd-4f3b-44cb-a6b8-eca10d8191fc admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.011187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.012026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.012510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.012838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.013595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:13.013949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.015871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.016882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 794/1568] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] POST /identity/v3/projects => generated 375 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.017708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.030678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 775/1569] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:13.031593 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.043622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbffe31-9971-49ca-8af5-eebcaabefeb1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.085870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.090742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caec772-3413-4677-83f3-29bc3dc26b97 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.096477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.097098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.097377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.097651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.098324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.100452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.105061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.105671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.105948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.106248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.106885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:13.107183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.109042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.111698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 795/1570] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.112626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.129500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 776/1571] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:13.129723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121f9ef7-546a-4200-901c-7d5594a5668e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.129850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.130585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.130585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.130585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.130585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.131057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:13.131057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.132105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.133928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcdbd7-f66f-49a9-b463-6691d031861d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.138996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.139592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.139880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.140134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.140765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:13.141063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.142704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.146562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 796/1572] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:13.150863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.159524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 777/1573] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:13.160338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.161998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e44b99-f135-4fa4-97d2-e8c11f156d00 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.166583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.167849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.168138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.168414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.169045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.171054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01bbf19-4da6-4d66-8dc1-73e176cab412 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.171181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.176089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.177822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.178140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.178415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.179033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.180908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.198200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 797/1574] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 389 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.199242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.212611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 778/1575] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.213589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.222423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.266380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b646496-df8e-4e60-81d1-a008b38eebad admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.282296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07679f4f-ba7c-4b06-8861-565ed2d532ea admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.287556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.288178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.288461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.288744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.289381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.291417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.320627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 779/1576] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 383 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.322139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.327862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.328475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.328754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.329022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.329647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.331730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.357832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 798/1577] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 294 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.357832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.366536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755c552-8e75-4c13-8808-2a93bcdc5e4c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.388712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03b7e62-be1b-4ea2-a76f-4b758fa9d369 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.393206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.393816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.394128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.394402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.395040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.396784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.409758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 780/1578] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.409758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.417349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0012170-bb5e-4cf7-ae00-f0aa8dfee8f5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.420036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.422288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.422988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.423278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.423614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.424174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.426194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.429872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.430517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.430799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.431072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.431708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.433598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.455498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 781/1579] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.456309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.464877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 799/1580] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 393 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.465894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.520584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24096e85-3bd4-4293-95ac-ea85f9898d76 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.529404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.529997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee28508a-9aae-4dc4-89d1-9f41976c8849 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.535542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.536509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.536802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.537061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.537693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.539649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.560948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 800/1581] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 295 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.560948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.572220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c565e-40b4-45fa-873a-af488a96960b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.576967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.577542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.577816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.578109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.578720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.580632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.580925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.581535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.581845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.582318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.583440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.584850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.602093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 801/1582] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 289 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.603257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.616837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 782/1583] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/projects => generated 405 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.617024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6d823b-62ea-48ad-b969-69d6ee1c6fc4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.617156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.670818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.671438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.671712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.671987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.672611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.674631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.675613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8f303f-faf9-4734-bb65-765637a82324 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.681048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.681683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.681957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.682261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.682897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.684480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.699256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 783/1584] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.699256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.701479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 802/1585] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.702635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.707022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807b1a8-8b64-4a93-992b-51dde18abc1d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.711579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.712250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.712525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.712794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] PATH_INFO: `/v3/projects/115fe2408cba49bb8b84af24fc8ce604/users/1903d1aaa3444181ac7b63234f4e8970/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.713565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313303da-b5d1-4741-b010-a7e004a4a57d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.718163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.718759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.719037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.719307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.719928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.721861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.728651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] RBAC: Authorizing `identity:create_grant(project_id=115fe2408cba49bb8b84af24fc8ce604, user_id=1903d1aaa3444181ac7b63234f4e8970, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.731359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.744781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 803/1586] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 284 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.745534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 784/1587] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:13 2022] PUT /identity/v3/projects/115fe2408cba49bb8b84af24fc8ce604/users/1903d1aaa3444181ac7b63234f4e8970/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:13.745737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.746212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.756097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428ba628-5ed3-4ca0-86b2-143cdeaa9b39 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.778752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:INSERT=13 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.813888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eef846d-5bef-499a-9f2f-cc5cc3fa6219 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.816642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.817260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.817534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.817805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.818491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.818611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.819194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.819479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.819754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.820399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.820502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.821993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.831897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 785/1588] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.832717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.842564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe8d13f-9cc9-4279-839c-58450ca125cc admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.846044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 804/1589] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.846044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.850747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.851356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.851635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.851910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.852548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.854162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.854271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cc4b6-81f1-4457-b589-387eeecdd906 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.855487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:13.859454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.860047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.860330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.860607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.861229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.863153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.867351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 786/1590] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.868403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.878467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0191a19-295b-495f-a9d6-40fc4095957f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.883628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.884262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.884535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.884805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] PATH_INFO: `/v3/projects/54ffd40154a249b892dfcf49adffa6c1/users/44410a2bf56548c3ba5884f436925d7a/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.885938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 805/1591] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/users => generated 303 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:13.886830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.896403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51f3e5f-f863-4930-9cff-4f8e0dc41573 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.901002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.901479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=54ffd40154a249b892dfcf49adffa6c1, user_id=44410a2bf56548c3ba5884f436925d7a, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.901941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.901941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.902165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.902770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.904145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.904403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a08aa-12ab-45b5-932d-120063b032e7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.915130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 806/1592] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:13.916287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.916576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.916850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.920380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 787/1593] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:13 2022] PUT /identity/v3/projects/54ffd40154a249b892dfcf49adffa6c1/users/44410a2bf56548c3ba5884f436925d7a/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:13.920380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:13.935760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:13.936978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:13.982636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0025dd3b-83f6-4953-a81b-10a0f96d0c9d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:13.987607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:13.988228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:13.988504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:13.988845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:13.990096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:13.991880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:13.996818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 807/1594] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:13.997621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.002196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 788/1595] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.003078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.007549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2f477f-3b2d-4688-9570-1af4c3f61110 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.012273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.012755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c981ba77-1f13-484f-aa43-5a5524e329a2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.012860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.013141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.013411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.014080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.015897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.017499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.018141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.018429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.018702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.019367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.021157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.025813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 808/1596] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.026698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.034153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 789/1597] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.035029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.037506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef7973-3cf4-4d48-9e2d-33667cb43d00 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.041992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.042660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.043002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.043346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.043970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.045209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318c3e4d-3014-4f4e-9e4b-f1ac48e80194 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.045529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.050411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.051520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.052424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.052695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.053482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.055058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.059014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 809/1598] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.059014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.068452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 790/1599] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.068452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.068714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bf471-888a-4a7e-85b8-238555c7650f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.071111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.071690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.071963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.072234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.072858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.074440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.075937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fed186-0cd1-4d57-93a6-049afe31c2af admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.080370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.080938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.081218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.081486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.082132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.084197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.085342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 810/1600] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.086191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.095423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4540d-1aca-423d-be04-ab33407fe42f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.099420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 791/1601] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.099420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.102293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.102893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.103156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.103422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.104050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.105603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.115759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 811/1602] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.116481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.125866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fff56f-2ee5-49a6-b9b4-e0519412d3ad admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.130507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.131130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.131407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.131680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] PATH_INFO: `/v3/projects/312745d2566343faa50fa7c6f06483f1/users/fc81231c965947ecaced8013076edafc/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.146279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] RBAC: Authorizing `identity:create_grant(project_id=312745d2566343faa50fa7c6f06483f1, user_id=fc81231c965947ecaced8013076edafc, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.148889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.156308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41111a83-262b-4b54-ab23-f7bb2cd8139d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.163835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 812/1603] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/312745d2566343faa50fa7c6f06483f1/users/fc81231c965947ecaced8013076edafc/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.164718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.184304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:14.206321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.206930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.207207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.207481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.212307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:14.228961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c85da2a-d7d3-4eea-ba35-bdd8355df29a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.233770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.234413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.234691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.234963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.235596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.237175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.249511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 813/1604] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.249511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.262698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda345a-980b-45a3-8435-4ca0bae54bb6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.267923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.268723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.269185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.269607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] PATH_INFO: `/v3/projects/ee4896d989eb4479a566fab8448de5fd/users/12e97df696b4444db047489b59e19feb/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.277162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.278672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.286858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee4896d989eb4479a566fab8448de5fd, user_id=12e97df696b4444db047489b59e19feb, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.288749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 792/1605] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:14.289520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.289683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.299556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd2c2f-af52-41e0-83f8-7ad186b31006 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.304348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 814/1606] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/ee4896d989eb4479a566fab8448de5fd/users/12e97df696b4444db047489b59e19feb/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.305374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.376570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.377212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.377488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.378025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.378718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.380200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef8d58-ab7f-4bb9-9160-335c54ad834c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.380361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.387082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.388180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.388632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.388905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] PATH_INFO: `/v3/projects/2adeeab71bd444d290f405908fb221fd/users/0fe86aa27b1741e38732e0e97834a4f5/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.396042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 793/1607] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:14.396042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.405825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2adeeab71bd444d290f405908fb221fd, user_id=0fe86aa27b1741e38732e0e97834a4f5, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.407062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6342309-928b-43e9-b9ce-ede2351416c0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.408564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.413003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.413692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.413988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.414433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] PATH_INFO: `/v3/projects/54ffd40154a249b892dfcf49adffa6c1/users/44410a2bf56548c3ba5884f436925d7a/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.416648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54ffd40154a249b892dfcf49adffa6c1, user_id=44410a2bf56548c3ba5884f436925d7a, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.419701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e325272-4d45-4d15-b7af-add8308e33b7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.433835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 815/1608] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/2adeeab71bd444d290f405908fb221fd/users/0fe86aa27b1741e38732e0e97834a4f5/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.433835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.437273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 794/1609] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/54ffd40154a249b892dfcf49adffa6c1/users/44410a2bf56548c3ba5884f436925d7a/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.442273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.442669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.442994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.464265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:14.466104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:14.518066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02de196-9870-4e4f-8b97-771ac6781314 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.536563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:14.548211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 795/1610] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:14.549501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.561316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad8ee5e-8994-40f0-b4b1-659a94d19e8c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.566764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.567981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.568262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.568538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] PATH_INFO: `/v3/projects/f1f49235fbe64b79bfef2cf828993c56/users/347e262907da4a83a197aaf42cecf51d/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.587960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1f49235fbe64b79bfef2cf828993c56, user_id=347e262907da4a83a197aaf42cecf51d, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.590733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a27500d-a6e6-424b-9bfe-68c66fe6acbd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.593657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.594499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.595336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] PATH_INFO: `/v3/projects/bd412080ef1e42b48b0014e3d1399915/users/91f37a04e7e44b8a8f79f23199b029aa/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.601456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:14.607534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 796/1611] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/f1f49235fbe64b79bfef2cf828993c56/users/347e262907da4a83a197aaf42cecf51d/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.609154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7b284-0fed-4b9f-9891-8770c161db31 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.609737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7b284-0fed-4b9f-9891-8770c161db31 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.610572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe7b284-0fed-4b9f-9891-8770c161db31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.627500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd412080ef1e42b48b0014e3d1399915, user_id=91f37a04e7e44b8a8f79f23199b029aa, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.631710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.647232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3fe7b284-0fed-4b9f-9891-8770c161db31 None None] MFA Rules not processed for user `12e97df696b4444db047489b59e19feb`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:14.649011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3fe7b284-0fed-4b9f-9891-8770c161db31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:14.652736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 816/1612] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/bd412080ef1e42b48b0014e3d1399915/users/91f37a04e7e44b8a8f79f23199b029aa/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.653867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.734610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e945f-35a8-4e50-b97d-16aedfb80bc3 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.735081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 797/1613] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:14.736713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.736996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.737276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.742844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:14.743659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.744483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.744773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] PATH_INFO: `/v3/projects/1cc133d79c324a2ba6cae360c161ccea/users/7a60ecd859a54236bdfa7351477ad319/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.763766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1cc133d79c324a2ba6cae360c161ccea, user_id=7a60ecd859a54236bdfa7351477ad319, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:14.766488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e97ce5-c674-4965-b776-0e8754c4e26b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:14.769803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] MFA Rules not processed for user `0fe86aa27b1741e38732e0e97834a4f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:14.771676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:14.782439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 817/1614] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:14 2022] PUT /identity/v3/projects/1cc133d79c324a2ba6cae360c161ccea/users/7a60ecd859a54236bdfa7351477ad319/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:14.784109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6012253-947a-411c-9219-cae71b8a6251 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.784555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6012253-947a-411c-9219-cae71b8a6251 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.784971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6012253-947a-411c-9219-cae71b8a6251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.815568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a6012253-947a-411c-9219-cae71b8a6251 None None] MFA Rules not processed for user `44410a2bf56548c3ba5884f436925d7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:14.816832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a6012253-947a-411c-9219-cae71b8a6251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:14.854164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 798/1615] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:14.867440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:14.893863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 818/1616] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:14.893863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075276be-db32-46bc-a7e9-51fa8f5a165f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.893863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075276be-db32-46bc-a7e9-51fa8f5a165f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.893863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075276be-db32-46bc-a7e9-51fa8f5a165f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:14.919044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-075276be-db32-46bc-a7e9-51fa8f5a165f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:14.920435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-075276be-db32-46bc-a7e9-51fa8f5a165f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:14.927140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:14.957041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd8a942-7e5e-4b48-b411-85fa7fd6d9f1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:14.994255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 819/1617] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:14.996427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963cdf0d-ab09-433f-bb10-0b7670608767 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:14.996718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963cdf0d-ab09-433f-bb10-0b7670608767 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:14.996996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963cdf0d-ab09-433f-bb10-0b7670608767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.002961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:15.026143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.026860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.027208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.027582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.029647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-963cdf0d-ab09-433f-bb10-0b7670608767 None None] MFA Rules not processed for user `91f37a04e7e44b8a8f79f23199b029aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:15.031035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-963cdf0d-ab09-433f-bb10-0b7670608767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:15.039584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.041568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.054284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 799/1618] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:15.055342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.066139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb37e3-5184-4c12-9ae8-a92118589191 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.072255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.073308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.073435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.073762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.074574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.077496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034c7a14-a0e6-428c-a89a-2b642984233f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.113506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 800/1619] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/projects => generated 379 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:15.120077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.120509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.121111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.135383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 820/1620] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:15.135383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b560f-74b7-41ef-b372-efc50d495900 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.135383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b560f-74b7-41ef-b372-efc50d495900 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.135383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906b560f-74b7-41ef-b372-efc50d495900 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.136005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 821/1621] 10.222.0.46 () {54 vars in 960 bytes} [Fri Oct 28 11:01:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:15.136924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906b560f-74b7-41ef-b372-efc50d495900 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.166544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] MFA Rules not processed for user `7a60ecd859a54236bdfa7351477ad319`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:15.167995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:15.226060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906b560f-74b7-41ef-b372-efc50d495900 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.232149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.233017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.233708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.234005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.266347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 801/1622] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:15.266347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.315091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:15.319738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.322764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.337672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 822/1623] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:15.338992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.365009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43bb0d7-391e-46d4-b81d-ed3b18198d0f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.373128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.375864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.376004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.376374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.424469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b9e9c6-4c3a-4812-a8e2-372ba54bee66 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.432690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.436455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.436600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.437109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.437858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.443416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b7f1ec-ff86-4489-a4da-928bec5f26a6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.488454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 823/1624] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/users => generated 296 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:15.493059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.493391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.493652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.515244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:15.518359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.522685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.544450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 802/1625] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:15.546211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.565563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9107178a-8132-4efd-8703-b966619bf998 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.572338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.572338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.572649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.573323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.576806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:15.578503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:15.683844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.686126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.701345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 803/1626] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:15 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:15.702336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.717532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5d4d2e-c9ab-4598-ac34-9f8d817e59fe service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.725238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 824/1627] 10.222.0.46 () {58 vars in 1057 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:15.726184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.744876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e15805-09ea-475f-b1f6-191dddef5576 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.754969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.756092 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.756734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.757020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.757299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.757949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.758747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.759915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.760591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.761670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.764261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.769421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.790994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 825/1628] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:15 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:15.791953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.813473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fc858c-f3af-4103-83ac-11c0d083214e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:15.830601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:15.831332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:15.831621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:15.831897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:15.855712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 804/1629] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/projects => generated 405 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:15.855712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.937399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:15.954990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:15.957747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:15.981515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 826/1630] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:15.982954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:15.993029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4aba2-bdbb-424f-b0d2-d7b285d210bc admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.003494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.004107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.004385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.004698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] PATH_INFO: `/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1/users/d680df1fe14f42aebdd70a1c10f768bd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.006619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df275ad0-41e8-4af5-8922-8e8a58d5e43c service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.019471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.020163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.020574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.020849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.021559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.025094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.038353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0dbbe5d08fcf4739acc3c00d75b2c5b1, user_id=d680df1fe14f42aebdd70a1c10f768bd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.043033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.077868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 805/1631] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:15 2022] PUT /identity/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1/users/d680df1fe14f42aebdd70a1c10f768bd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:16.078552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 827/1632] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:15 2022] POST /identity/v3/users => generated 309 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:16.079791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.082120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.102846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483e6ea-3a00-4e2e-a09a-a7a8c9e7181b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.226449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:16.243250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbdbd5-ecbd-4b39-ae92-6e147ed9e6c2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.254624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.256052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.257557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.258004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.258968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.260886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.261570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.263939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.266403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.270916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.271359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.273000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.285311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 828/1633] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:16 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:16.290955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.297552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 806/1634] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:16 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:16.302481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.320202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003773d-9cf3-4480-9621-065e8546671c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.323448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b0f708-111c-44ca-9554-e0d08f5fa639 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.336458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.337400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.337764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.338182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] PATH_INFO: `/v3/projects/4c7d76d25cb843579e5a12ff315873a2/users/1114243702a14882a4442da75a4db32e/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.349413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.350261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.350499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.350785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] PATH_INFO: `/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1/users/d680df1fe14f42aebdd70a1c10f768bd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.353293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0dbbe5d08fcf4739acc3c00d75b2c5b1, user_id=d680df1fe14f42aebdd70a1c10f768bd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.356246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59bd47e-e0ea-4054-a8f9-7d1c36ec407f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.385281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 807/1635] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:16 2022] PUT /identity/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1/users/d680df1fe14f42aebdd70a1c10f768bd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:16.388511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.388938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.391147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.392848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c7d76d25cb843579e5a12ff315873a2, user_id=1114243702a14882a4442da75a4db32e, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.396786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.429163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 829/1636] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:16 2022] PUT /identity/v3/projects/4c7d76d25cb843579e5a12ff315873a2/users/1114243702a14882a4442da75a4db32e/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:16.430650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.518924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] MFA Rules not processed for user `d680df1fe14f42aebdd70a1c10f768bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:16.518924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:16.662934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b735a0e2-e563-43e1-b433-b78b8ec142e5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.668328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 808/1637] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:16 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:16.680962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.696515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.697166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.697440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.697711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.698398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.700028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67190ad-566f-4fec-8105-d170ed746123 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.722211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 830/1638] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:16 2022] GET /identity/v3/roles => generated 1420 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:16.722211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e68569-74f1-4157-8807-9814a2be026e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.722438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e68569-74f1-4157-8807-9814a2be026e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.724888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e68569-74f1-4157-8807-9814a2be026e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.726303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 831/1639] 10.222.0.46 () {54 vars in 912 bytes} [Fri Oct 28 11:01:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:16.731757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e68569-74f1-4157-8807-9814a2be026e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:16.756557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e68569-74f1-4157-8807-9814a2be026e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.797224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.798276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.799859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.802503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] PATH_INFO: `/v3/projects/4c7d76d25cb843579e5a12ff315873a2/users/1114243702a14882a4442da75a4db32e/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.807239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5835c52-9278-48a3-842b-f263d51fe7e8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:16.808243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c7d76d25cb843579e5a12ff315873a2, user_id=1114243702a14882a4442da75a4db32e, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.811610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:16.812586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.812931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.813271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.817884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98c7cf2-d633-45b9-a27f-70c1bac8be62 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.826216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:16.830512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e2e6f4-31b1-4876-b1a4-935a42d90ebf service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:16.833382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 832/1640] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:16 2022] PUT /identity/v3/projects/4c7d76d25cb843579e5a12ff315873a2/users/1114243702a14882a4442da75a4db32e/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:16.835029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.835523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.836020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.847522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 809/1641] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:16 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:16.849075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:16.849466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:16.849810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:16.864608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:16.866214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:16.891950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] MFA Rules not processed for user `1114243702a14882a4442da75a4db32e`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:16.894184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:16.925178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:17.002489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 833/1642] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:01:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:17.028903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:17.031853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:17.076019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 810/1643] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:16 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:17.076019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:17.176064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295735c5-e139-49d4-b8bd-55ec57310713 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:17.188952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:17.191161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:17.191161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:17.191161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:17.196866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd1d0ef-3302-498a-bd7d-acc4c3c707c2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:17.205770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:17.206447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:17.207475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:17.207610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:17.207765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:17.207968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:17.223183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 834/1644] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:17.292552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:17.294485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:17.307963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 811/1645] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:17.567288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:17.580959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e04f4f3-909c-4e59-809b-c13d698e5fa3 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:17.592383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:17.593092 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:17.593322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:17.593593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:17.623132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:17.624690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d7dcf8-ca17-422d-a0ba-c5585033f3a6 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:17.648710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 835/1646] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:17 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:18.781485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:18.909992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cec7dd-51be-43a0-82be-47b6b876a324 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:18.917858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:18.919302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:18.920425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:18.920718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] PATH_INFO: `/v3/users/7a60ecd859a54236bdfa7351477ad319` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:18.926261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a60ecd859a54236bdfa7351477ad319)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:18.931831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:18.943989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45f461-3b58-494d-ab6c-94cc315bcf33 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:18.944553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45f461-3b58-494d-ab6c-94cc315bcf33 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:18.944553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec45f461-3b58-494d-ab6c-94cc315bcf33 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:18.946599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 836/1647] 10.222.0.46 () {54 vars in 960 bytes} [Fri Oct 28 11:01:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:18.955022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bcc7c-85c7-4eba-b979-3ad2fdbe820e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:18.955452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bcc7c-85c7-4eba-b979-3ad2fdbe820e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:18.955759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bcc7c-85c7-4eba-b979-3ad2fdbe820e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.066255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ca1bcc7c-85c7-4eba-b979-3ad2fdbe820e None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:19.067590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ca1bcc7c-85c7-4eba-b979-3ad2fdbe820e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:19.104045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 837/1648] 10.222.0.46 () {58 vars in 1057 bytes} [Fri Oct 28 11:01:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:19.110754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0303a-16a4-4af7-879f-d56b262fe141 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:19.111215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0303a-16a4-4af7-879f-d56b262fe141 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:19.111696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0303a-16a4-4af7-879f-d56b262fe141 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.113326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 838/1649] 10.222.0.46 () {54 vars in 912 bytes} [Fri Oct 28 11:01:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:19.119768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f8280f-3c0f-4164-aec0-502f84b1b02e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:19.121524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f8280f-3c0f-4164-aec0-502f84b1b02e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:19.122553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f8280f-3c0f-4164-aec0-502f84b1b02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.164317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a60ecd859a54236bdfa7351477ad319'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:19.168085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a60ecd859a54236bdfa7351477ad319'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:19.173992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:19.195761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a60ecd859a54236bdfa7351477ad319'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:19.196716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a60ecd859a54236bdfa7351477ad319'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:19.199489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a4f8280f-3c0f-4164-aec0-502f84b1b02e None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:19.206907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a4f8280f-3c0f-4164-aec0-502f84b1b02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:19.213786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a60ecd859a54236bdfa7351477ad319'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:19.245836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 839/1650] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:01:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:19.284138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 812/1651] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:18 2022] DELETE /identity/v3/users/7a60ecd859a54236bdfa7351477ad319 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:19.286942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:19.287232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:19.287510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.324263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:19.335233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:19.340765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:19.342639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:19.519687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e853324-7dc1-4ffb-8e62-4fee85695276 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:19.522835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 840/1652] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:19.529141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:19.529882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:19.530293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:19.530749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.560918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:19.562722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:19.691418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:19.745873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:19.753357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:19.753876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f7771e-23db-4085-87ac-92de25b3f56c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:19.772028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14458070-7924-4435-973c-9a4615358491 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:19.772820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14458070-7924-4435-973c-9a4615358491 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:19.773301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14458070-7924-4435-973c-9a4615358491 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:19.773834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14458070-7924-4435-973c-9a4615358491 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:19.792500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 813/1653] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:19.801945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14458070-7924-4435-973c-9a4615358491 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:19.804886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14458070-7924-4435-973c-9a4615358491 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:19.834117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 841/1654] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:22.316458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:22.319271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:22.323978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a245f3a-0edd-4c02-8ed3-8d4dd093355c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:22.325508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a245f3a-0edd-4c02-8ed3-8d4dd093355c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:22.325950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a245f3a-0edd-4c02-8ed3-8d4dd093355c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:22.352612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5550b8ef-a0bf-41ab-8c71-ba1502596852 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:22.358943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:22.360355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:22.360883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:22.361320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] PATH_INFO: `/v3/projects/1cc133d79c324a2ba6cae360c161ccea` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:22.363001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1cc133d79c324a2ba6cae360c161ccea)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:22.364599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: WARNING py.warnings [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:22.364599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: warnings.warn(msg) Oct 28 11:01:22.364599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: Oct 28 11:01:22.365818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:22.391138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6a245f3a-0edd-4c02-8ed3-8d4dd093355c None None] MFA Rules not processed for user `1903d1aaa3444181ac7b63234f4e8970`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:22.404523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6a245f3a-0edd-4c02-8ed3-8d4dd093355c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:22.492483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1cc133d79c324a2ba6cae360c161ccea', 'request_id': 'req-2f288fbe-5b51-437d-b889-4686f55e9440'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.502713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1cc133d79c324a2ba6cae360c161ccea', 'request_id': 'req-2f288fbe-5b51-437d-b889-4686f55e9440'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.510586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 842/1655] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:22.517288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1cc133d79c324a2ba6cae360c161ccea', 'request_id': 'req-2f288fbe-5b51-437d-b889-4686f55e9440'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.523107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:INSERT=1,keystone:DELETE=9,keystone:SELECT=90 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:22.523478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:22.524396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:22.524709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:22.537371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] The token cache is being invalidate because project 1cc133d79c324a2ba6cae360c161ccea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:22.537935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.539303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.541866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 814/1656] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:22 2022] DELETE /identity/v3/projects/1cc133d79c324a2ba6cae360c161ccea => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:22.555692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:22.564738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:22.578597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] MFA Rules not processed for user `1903d1aaa3444181ac7b63234f4e8970`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:22.583801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:22.711151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f288fbe-5b51-437d-b889-4686f55e9440 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:22.733900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:22.734640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:22.734925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:22.735204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] PATH_INFO: `/v3/users/1114243702a14882a4442da75a4db32e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:22.736622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1114243702a14882a4442da75a4db32e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:22.738743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:22.753469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 843/1657] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:22.766575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:22.768572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:22.845890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1114243702a14882a4442da75a4db32e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.846744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1114243702a14882a4442da75a4db32e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.874680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1114243702a14882a4442da75a4db32e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.875418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1114243702a14882a4442da75a4db32e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.898977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1114243702a14882a4442da75a4db32e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:22.960742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 815/1658] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:22 2022] DELETE /identity/v3/users/1114243702a14882a4442da75a4db32e => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:23.035129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32935de-6f7a-476a-b6bb-b6d3038e8184 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:23.053508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:23.170616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:23.172270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:23.172816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:23.173177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:23.180082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:23.358503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:23.360814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f465f1-f7e3-4686-a53d-1945e5a33a6d service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:23.371891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 844/1659] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:22 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 606 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:24.563357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:24.565742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:24.598985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4d8a9-74cf-40ab-8d5e-039fd14434e0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:24.600391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4d8a9-74cf-40ab-8d5e-039fd14434e0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:24.600740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4d8a9-74cf-40ab-8d5e-039fd14434e0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:24.603064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 845/1660] 10.222.0.46 () {54 vars in 912 bytes} [Fri Oct 28 11:01:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:24.609445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:24.609940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:24.611005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:24.703177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:24.705118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:24.770742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35333898-a38c-46f3-a478-fd8604da5ac4 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:24.800467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:24.802705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:24.802705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:24.804550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] PATH_INFO: `/v3/projects/4c7d76d25cb843579e5a12ff315873a2` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:24.820728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c7d76d25cb843579e5a12ff315873a2)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:24.828008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:24.872571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:24.919896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 846/1661] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:01:24 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:24.935789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:24.937484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:24.977048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b40cd9-260b-4adc-82a9-82fc810501f7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:24.985455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c7d76d25cb843579e5a12ff315873a2', 'request_id': 'req-39f07e07-de6c-48a6-8167-373c3de6b3f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:24.995083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c7d76d25cb843579e5a12ff315873a2', 'request_id': 'req-39f07e07-de6c-48a6-8167-373c3de6b3f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:25.016287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c7d76d25cb843579e5a12ff315873a2', 'request_id': 'req-39f07e07-de6c-48a6-8167-373c3de6b3f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:25.048507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] The token cache is being invalidate because project 4c7d76d25cb843579e5a12ff315873a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:25.049144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:25.050684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39f07e07-de6c-48a6-8167-373c3de6b3f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:25.052782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 816/1662] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:24 2022] DELETE /identity/v3/projects/4c7d76d25cb843579e5a12ff315873a2 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:25.060755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.061785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.062429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.110088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:25.111805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:25.113103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:25.118583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.120331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.120331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.120494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.128071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:25.149037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.154954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ef6724-623b-470b-8dd8-9f9f8ccbd6a8 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.186407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 847/1663] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:25.210165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 817/1664] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:25.215048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.215197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.215748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.241802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:25.244930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:25.270020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 848/1665] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:25.273104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.294996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221c923-1c60-458c-80dc-b01765a3eb86 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.305425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.306691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.306977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.307299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.307986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:25.308371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.310122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.326640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 818/1666] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:25.331185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.342798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d9b9c-d502-4c03-84c3-66e79bd58b6b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.357199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.358629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.359276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.360175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.361763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.364403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.443577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 849/1667] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:25 2022] POST /identity/v3/projects => generated 405 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:25.450529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.464655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358b827b-b2ef-4520-99b7-17106809604a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.484214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.487233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:25.625074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97e3eb-d900-43bd-b8bd-fbcc6ddc13b1 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.631481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.632680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.633158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.633158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.634178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.634400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.634681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.635262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:25.636958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.641279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.681560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 819/1668] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:25 2022] POST /identity/v3/users => generated 303 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:25.687327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.700416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0407a341-48be-43b4-a8e1-7088d1e11d8f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.740096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.741099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.743021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.744204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.754955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.759452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.796767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.798265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.802722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 820/1669] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:25 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:25.803703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.828253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:25.828984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 850/1670] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:25 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:25.841604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc792bd-0335-4fbf-9661-bbe230d8889d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.848410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.849053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.850350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.851237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:25.851911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:25.855201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eb4b15-8238-4f03-bf3c-25faddaae5a7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:25.911083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 821/1671] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:25 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:25.911620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:25.929866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f364632c-9ddb-4db3-8a2f-287f0e376b8c service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:25.950622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:25.951395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:25.951894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:25.952243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] PATH_INFO: `/v3/projects/da70f15569f748bcbcad97944a01bda1/users/d53a2c443cdd4da2a8311cd50c9e7e2f/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:26.001179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da70f15569f748bcbcad97944a01bda1, user_id=d53a2c443cdd4da2a8311cd50c9e7e2f, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:26.009800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:26.047151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 851/1672] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:25 2022] PUT /identity/v3/projects/da70f15569f748bcbcad97944a01bda1/users/d53a2c443cdd4da2a8311cd50c9e7e2f/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:26.054254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:26.054779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:26.055120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:26.101380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:26.105079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:26.289478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 822/1673] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:26.458954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:26.461868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:26.584409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:26.586370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:26.625060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:26.633046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81366e4e-1779-47cb-8f7f-1089ded527f9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:26.640147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:26.641061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:26.641886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:26.642300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:26.663029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:26.664962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73bd44e-02eb-45e4-afca-2da867dd7f30 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:26.681907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 852/1674] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:26.792761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17398784-8a52-41b3-b0b5-9139df8937f4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:26.829035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:26.830300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:26.833772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:26.834101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:26.842682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:26.865465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:26.867329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:26.894739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 823/1675] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:27.384311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13ad27-1872-410e-8a00-5933352db63e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:27.384658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13ad27-1872-410e-8a00-5933352db63e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:27.384853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13ad27-1872-410e-8a00-5933352db63e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:27.437401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-fb13ad27-1872-410e-8a00-5933352db63e None None] MFA Rules not processed for user `fc81231c965947ecaced8013076edafc`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:27.439199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fb13ad27-1872-410e-8a00-5933352db63e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:27.576658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 853/1676] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:27 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:27.593863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:27.595576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:27.771129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe049805-ecd8-4f25-b5c2-029c69ce6e8c service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:27.783863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:27.787022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:27.787371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:27.787690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:27.811490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:27.813999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565dcebd-bf84-43ad-b4a1-5a0d791eefd9 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:27.826369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 824/1677] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:28.321753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:28.322376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:28.322376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:28.392318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] MFA Rules not processed for user `347e262907da4a83a197aaf42cecf51d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:28.394439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:28.637623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 854/1678] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:28 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:28.643713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca6722-104a-444a-b0af-e7f9b1085b53 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:28.644001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca6722-104a-444a-b0af-e7f9b1085b53 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:28.644328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca6722-104a-444a-b0af-e7f9b1085b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:28.700311 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f7ca6722-104a-444a-b0af-e7f9b1085b53 None None] MFA Rules not processed for user `347e262907da4a83a197aaf42cecf51d`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:28.704258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f7ca6722-104a-444a-b0af-e7f9b1085b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:28.732156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 825/1679] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:28 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:28.744231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:28.745851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:28.800048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd251a67-9a42-44af-9831-53e41887f7cc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:28.815854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:28.816729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:28.817537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:28.819367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:28.843674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:28.846731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:28.872151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 855/1680] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:28 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:29.688162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:29.689477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:29.690086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:29.700183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:29.782895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] MFA Rules not processed for user `d53a2c443cdd4da2a8311cd50c9e7e2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:29.787969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:30.035941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 826/1681] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:29 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:30.046767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:30.063076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c3a89-a0a1-44c7-bfb1-4fbdc55c11bc service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:30.075980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:30.076722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.077030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.077479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.078329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:30.081256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:30.135291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 856/1682] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:30 2022] POST /identity/v3/projects => generated 403 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:30.136075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:30.152713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487be25-15fd-4c2f-b33e-a74a23aad3f5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:30.309515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:30.311093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.311586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.312069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.312949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:30.315326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:30.367307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:30.375055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 827/1683] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:30 2022] POST /identity/v3/users => generated 308 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:30.382370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ea953c-3869-454d-99d9-c0a75b922ab7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:30.388770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:30.389914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.390426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.390875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.391712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:30.393911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d79c82a-cd19-4af7-a33d-9f29f96fe0ff admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:30.423243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 857/1684] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:30 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:30.434995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:30.470830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51381c6d-30f3-408e-962d-c8a349d07e98 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:30.486668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:30.487501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.488083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.489889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] PATH_INFO: `/v3/projects/4e51d2d6b1164469b399420f069935f7/users/719b96d0d250467dbc82c8ccf68cf2e2/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.552966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e51d2d6b1164469b399420f069935f7, user_id=719b96d0d250467dbc82c8ccf68cf2e2, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:30.557019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:30.599644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 828/1685] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:30 2022] PUT /identity/v3/projects/4e51d2d6b1164469b399420f069935f7/users/719b96d0d250467dbc82c8ccf68cf2e2/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:30.608887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.610155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.611098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.620241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:30.678170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] MFA Rules not processed for user `347e262907da4a83a197aaf42cecf51d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:30.679878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:30.708643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:30.778490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 858/1686] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:30.911970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:30.931043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0447bb-0b87-4595-a715-160ff0446e1b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:30.956500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:30.957136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:30.958354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:30.958354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:30.958881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:30.960496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471aa076-27f1-4b50-9070-7a84d9136211 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:30.999606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 829/1687] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:30 2022] GET /identity/v3/roles => generated 1420 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:31.003711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:31.024218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8906d0-5b38-444e-8fc4-d23e4f194334 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:31.037374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:31.038104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:31.038386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:31.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] PATH_INFO: `/v3/projects/4e51d2d6b1164469b399420f069935f7/users/719b96d0d250467dbc82c8ccf68cf2e2/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:31.041156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e51d2d6b1164469b399420f069935f7, user_id=719b96d0d250467dbc82c8ccf68cf2e2, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:31.043874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07cf81-cd9d-431b-a310-bceb5d744ba5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:31.077167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 859/1688] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:31 2022] PUT /identity/v3/projects/4e51d2d6b1164469b399420f069935f7/users/719b96d0d250467dbc82c8ccf68cf2e2/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:31.662023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826a667-2b88-43a0-a88b-36697cdd24e5 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:31.662587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826a667-2b88-43a0-a88b-36697cdd24e5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:31.663063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826a667-2b88-43a0-a88b-36697cdd24e5 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:31.665476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 830/1689] 10.222.0.46 () {54 vars in 912 bytes} [Fri Oct 28 11:01:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:31.675716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:31.676183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:31.676733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:31.729865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:31.733060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:31.797492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0e803-44d5-49de-9bf9-7fad3485f3af None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:31.798778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0e803-44d5-49de-9bf9-7fad3485f3af None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:31.799307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab0e803-44d5-49de-9bf9-7fad3485f3af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:31.844613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-5ab0e803-44d5-49de-9bf9-7fad3485f3af None None] MFA Rules not processed for user `fc81231c965947ecaced8013076edafc`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:31.848845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5ab0e803-44d5-49de-9bf9-7fad3485f3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:31.864151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 860/1690] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:01:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:31.921466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:31.923591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:31.944789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3a7c0c-ae3e-4242-a8d6-b903318489af None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:31.959824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:31.963423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:31.964709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:31.966776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:31.981923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 831/1691] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:31 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:32.012421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:32.017475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d05206e-f753-467a-affe-5385633a82ad service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:32.057849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 861/1692] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:33.261170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:33.263313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:33.265341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:33.319968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] MFA Rules not processed for user `719b96d0d250467dbc82c8ccf68cf2e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:33.331327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:33.470225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 832/1693] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:33 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:33.912686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46b91d-7ea8-4ff7-8779-33cf365c793a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:33.914196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46b91d-7ea8-4ff7-8779-33cf365c793a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:33.914620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46b91d-7ea8-4ff7-8779-33cf365c793a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:34.089122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ab46b91d-7ea8-4ff7-8779-33cf365c793a None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:34.096304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ab46b91d-7ea8-4ff7-8779-33cf365c793a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:34.250101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 862/1694] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:01:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:34.517221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:34.519191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:34.674615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc20d07-fa21-43cf-b563-e6c97b10d71e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:34.685340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:34.686530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:34.687016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:34.687444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:34.696828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:34.702823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:34.704821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:34.716151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 833/1695] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:01:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:01:34.867840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a75bb-042f-495e-9039-983103eb1eea None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:34.868514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a75bb-042f-495e-9039-983103eb1eea None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:34.868887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a75bb-042f-495e-9039-983103eb1eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:35.008755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3c0a75bb-042f-495e-9039-983103eb1eea None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:35.021186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3c0a75bb-042f-495e-9039-983103eb1eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:35.079855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 863/1696] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Oct 28 11:01:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:35.094702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:35.139787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57ff8be-e3f2-425c-9fe5-27b5c852e4b2 service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:35.151347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:35.153944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:35.154584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:35.156986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:35.230536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:35.238566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13601e31-549f-4cc1-8c6b-08bac59ae06a service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:35.291235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 834/1697] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:35.705396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e5212-f497-4fde-9ecf-056777fc4b50 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:35.706060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e5212-f497-4fde-9ecf-056777fc4b50 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:35.706532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2e5212-f497-4fde-9ecf-056777fc4b50 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:35.709428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 864/1698] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:01:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:35.724881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:35.725899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:35.726415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:35.771091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:35.778842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:35.926623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 835/1699] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:01:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:35.937226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2e5212-f497-4fde-9ecf-056777fc4b50 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:35.956379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2e5212-f497-4fde-9ecf-056777fc4b50 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:35.962861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:35.963792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:35.964143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:35.964562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:35.976059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:35.978988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0628949-539a-422c-aa63-7d21b9dcfe2c service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:35.997471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 865/1700] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:37.123558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:37.205259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631ebfc0-d2a0-4c12-84a3-bf1bbc55e72d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:37.212793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:37.213556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.213772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.214227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] PATH_INFO: `/v3/users/d53a2c443cdd4da2a8311cd50c9e7e2f` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.218733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d53a2c443cdd4da2a8311cd50c9e7e2f)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:37.220847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:37.279258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd53a2c443cdd4da2a8311cd50c9e7e2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.280103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd53a2c443cdd4da2a8311cd50c9e7e2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.289180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd53a2c443cdd4da2a8311cd50c9e7e2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.289846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd53a2c443cdd4da2a8311cd50c9e7e2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.297566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd53a2c443cdd4da2a8311cd50c9e7e2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.309158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:37.322154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 836/1701] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:37 2022] DELETE /identity/v3/users/d53a2c443cdd4da2a8311cd50c9e7e2f => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:37.517076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e7517-382f-4733-8bee-0e0282e621b0 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.518334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e7517-382f-4733-8bee-0e0282e621b0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.518630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e7517-382f-4733-8bee-0e0282e621b0 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.520538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 866/1702] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:01:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:37.530207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:37.532244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:37.551755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.552080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.552579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.565774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:37.595932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:37.597147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:37.642432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f1dbde-e57d-4d24-b420-0fea9108bb11 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:37.647898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:37.648541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.648815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.649084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] PATH_INFO: `/v3/projects/da70f15569f748bcbcad97944a01bda1` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.650384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] RBAC: Authorizing `identity:delete_project(project_id=da70f15569f748bcbcad97944a01bda1)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:37.652461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:37.681060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 867/1703] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:01:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:37.700936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:37.702989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:37.706128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da70f15569f748bcbcad97944a01bda1', 'request_id': 'req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.716902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da70f15569f748bcbcad97944a01bda1', 'request_id': 'req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.721418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c834914d-0c2b-4be4-92aa-51d54ecc97f7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:37.727091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da70f15569f748bcbcad97944a01bda1', 'request_id': 'req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.742893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] The token cache is being invalidate because project da70f15569f748bcbcad97944a01bda1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:37.743206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.743851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.745580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 837/1704] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:37 2022] DELETE /identity/v3/projects/da70f15569f748bcbcad97944a01bda1 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:37.749368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:37.750816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:37.810411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:37.811208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.811554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.812129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.814612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:37.828040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:37.829626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:37.835455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb7d2b0-4597-4c86-97e6-ce33accbf60c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:37.840330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 868/1705] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:37.840516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:37.841066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:37.841354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:37.841626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] PATH_INFO: `/v3/users/719b96d0d250467dbc82c8ccf68cf2e2` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:37.842806 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] RBAC: Authorizing `identity:delete_user(user_id=719b96d0d250467dbc82c8ccf68cf2e2)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:37.844722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:37.876337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:37.903619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '719b96d0d250467dbc82c8ccf68cf2e2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.904210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '719b96d0d250467dbc82c8ccf68cf2e2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.913209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '719b96d0d250467dbc82c8ccf68cf2e2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.913932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '719b96d0d250467dbc82c8ccf68cf2e2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.922295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-03687db5-91a4-4360-ad38-3578a79eb551 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '719b96d0d250467dbc82c8ccf68cf2e2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:37.944993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 838/1706] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:37 2022] DELETE /identity/v3/users/719b96d0d250467dbc82c8ccf68cf2e2 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:38.156144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.169768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ae465-b92f-49e8-aeb6-028938484323 service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.228966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.229601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.229884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.230190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] PATH_INFO: `/v3/projects/4e51d2d6b1164469b399420f069935f7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.231308 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e51d2d6b1164469b399420f069935f7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.233460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.257511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:38.282517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e51d2d6b1164469b399420f069935f7', 'request_id': 'req-697888c8-eb25-4d53-83e3-38f7775ceb51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:38.290000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e51d2d6b1164469b399420f069935f7', 'request_id': 'req-697888c8-eb25-4d53-83e3-38f7775ceb51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:38.296363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e51d2d6b1164469b399420f069935f7', 'request_id': 'req-697888c8-eb25-4d53-83e3-38f7775ceb51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:38.309288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] The token cache is being invalidate because project 4e51d2d6b1164469b399420f069935f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:38.309515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:38.310121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697888c8-eb25-4d53-83e3-38f7775ceb51 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:38.311612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 869/1707] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:38 2022] DELETE /identity/v3/projects/4e51d2d6b1164469b399420f069935f7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:38.316631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.317160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.317160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.337682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:38.339124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:38.402091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 839/1708] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:38.409760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.410099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.410939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.429855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:38.431232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:38.439114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 870/1709] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:38.443243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.452705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d90fa6f-044d-413f-86e9-01df211aaa96 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.457759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.458390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.458668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.458943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.459590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:38.459884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.461489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.473358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 840/1710] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:38.477330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.486640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade76cb-666b-438e-8c06-e9cd118f9ddf None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.491144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.491732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.492004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.492269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.492893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.494788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.523412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 871/1711] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:38.527586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.537894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bdd957-5bc2-484b-bbfe-3095cec52428 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.593722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.594179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.594461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.594732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.595371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.597302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.621327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 841/1712] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/users => generated 294 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:38.622928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.633839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa218659-f356-467b-b06e-879c76894cc9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.638727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.639330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.639603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.639870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.640505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.642102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.652647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 872/1713] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:38.656539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.666708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72e9f7-c8ec-4254-bcdf-a973b9ec53c2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.671343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.671947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.672230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.672507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.673152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.674797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8effec36-e99c-483c-9ae0-1f833191abe0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.685276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 842/1714] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:38.689549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.699963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee2a3f-1c96-42bb-9f1a-dbe506b8a110 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.704780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.705427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.705706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.705996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] PATH_INFO: `/v3/projects/b9bbbd267713429e9a7f295da3b6209e/users/a3cbba7f906f4389bee2fcc905c63b97/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.720786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9bbbd267713429e9a7f295da3b6209e, user_id=a3cbba7f906f4389bee2fcc905c63b97, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.723449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.736359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 873/1715] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:38 2022] PUT /identity/v3/projects/b9bbbd267713429e9a7f295da3b6209e/users/a3cbba7f906f4389bee2fcc905c63b97/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:38.746450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.746732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.747003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.767314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:38.775954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] MFA Rules not processed for user `a3cbba7f906f4389bee2fcc905c63b97`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:38.777660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:38.847036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 843/1716] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:38.851616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.914113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbddfb-aa4c-4279-897d-6b65800d817c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:38.918819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:38.919437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:38.919721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:38.920001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:38.920653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:38.922606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:38.953450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 874/1717] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:38.954641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:38.965592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba5863-3f09-40aa-b13e-bca2f8bc8a7a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.025008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.025632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.025911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.026209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.026841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.029250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.062653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 844/1718] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:38 2022] POST /identity/v3/users => generated 300 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:39.068285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.084718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d33e9a-f405-4c7f-aeb9-f7562d0d7459 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.090363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.091057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.091348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.091950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.092991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.094794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.110859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 875/1719] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:39.116065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.126667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84a906-6bf6-43ad-a87b-c7930762fab3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.134701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.135540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.135822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.136239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] PATH_INFO: `/v3/projects/c1137d4ebee94020aa5e0152a1b0205c/users/540f94703ee9457bb9a46ead2592b3e3/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.153755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1137d4ebee94020aa5e0152a1b0205c, user_id=540f94703ee9457bb9a46ead2592b3e3, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.166416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.200558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 845/1720] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:39 2022] PUT /identity/v3/projects/c1137d4ebee94020aa5e0152a1b0205c/users/540f94703ee9457bb9a46ead2592b3e3/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:39.204588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.225942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e82ce6-0f67-4cc8-b73c-3a69041c3ffe admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.259245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:39.310533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.311148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.311428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.311698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.312341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.313922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0a9b4-f3eb-415e-a763-454782aa5f7f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.328708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 876/1721] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:39.330330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.342435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56da2f9-570e-479b-9f81-85c55df2363b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.366365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.367653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.367941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.368292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] PATH_INFO: `/v3/projects/c1137d4ebee94020aa5e0152a1b0205c/users/540f94703ee9457bb9a46ead2592b3e3/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.372428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1137d4ebee94020aa5e0152a1b0205c, user_id=540f94703ee9457bb9a46ead2592b3e3, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.375202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.389005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 846/1722] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:39 2022] PUT /identity/v3/projects/c1137d4ebee94020aa5e0152a1b0205c/users/540f94703ee9457bb9a46ead2592b3e3/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:39.400544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.400881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.401167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.436995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] MFA Rules not processed for user `540f94703ee9457bb9a46ead2592b3e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:39.438423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:39.522131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 877/1723] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:39 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:39.533279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.536073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:39.604843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e8d434-df9e-4efc-a114-a8fc6a0a6b48 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.608728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.609340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.609617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.609897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.616794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:39.619025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.620978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6c2d20-37da-4c3e-9030-e47a13b1e7c5 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.631130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 847/1724] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:39 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:39.696584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:39.759574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44984e63-1a59-4dc4-b5f8-5ff9bf1d2ab3 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:39.767055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:39.768117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.768593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.768936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] PATH_INFO: `/v3/users/a3cbba7f906f4389bee2fcc905c63b97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.770815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3cbba7f906f4389bee2fcc905c63b97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:39.771847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: WARNING py.warnings [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 11:01:39.771847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: warnings.warn(msg) Oct 28 11:01:39.771847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: Oct 28 11:01:39.773058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:39.845289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:39.852119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3cbba7f906f4389bee2fcc905c63b97'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:39.862512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3cbba7f906f4389bee2fcc905c63b97'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:39.863272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3cbba7f906f4389bee2fcc905c63b97'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:39.864373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3cbba7f906f4389bee2fcc905c63b97'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:39.877108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3cbba7f906f4389bee2fcc905c63b97'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:39.911028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 878/1725] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:39 2022] DELETE /identity/v3/users/a3cbba7f906f4389bee2fcc905c63b97 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:39.915284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:39.915570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:39.915866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:39.938865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:39.940664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:40.032473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 848/1726] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:40.048927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:40.050442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:40.176428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fa25a0-5853-4b0e-9a2e-698ca9d796de admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:40.184350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:40.184350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:40.184350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:40.184350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:40.198180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:40.200058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:40.212271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 879/1727] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:40.796033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:40.797663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:40.853410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bd2195-8ba1-4bca-9558-4433d1b43c79 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:40.861070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:40.861916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:40.862230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:40.862500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] PATH_INFO: `/v3/projects/b9bbbd267713429e9a7f295da3b6209e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:40.864537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9bbbd267713429e9a7f295da3b6209e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:40.877932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:40.963310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9bbbd267713429e9a7f295da3b6209e', 'request_id': 'req-070463fc-ec20-4a31-a76c-b9ac8a08168a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:40.969538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9bbbd267713429e9a7f295da3b6209e', 'request_id': 'req-070463fc-ec20-4a31-a76c-b9ac8a08168a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:40.976977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9bbbd267713429e9a7f295da3b6209e', 'request_id': 'req-070463fc-ec20-4a31-a76c-b9ac8a08168a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:40.993754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] The token cache is being invalidate because project b9bbbd267713429e9a7f295da3b6209e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:40.994150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:40.994713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:40.996550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 849/1728] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:40 2022] DELETE /identity/v3/projects/b9bbbd267713429e9a7f295da3b6209e => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:41.000453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:41.010618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8248bdc-ff03-4c16-aa57-c8e3b661df1a service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:41.092248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:41.092891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:41.093168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:41.093468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] PATH_INFO: `/v3/users/540f94703ee9457bb9a46ead2592b3e3` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:41.094623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] RBAC: Authorizing `identity:delete_user(user_id=540f94703ee9457bb9a46ead2592b3e3)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:41.096621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:41.166876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '540f94703ee9457bb9a46ead2592b3e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.179880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '540f94703ee9457bb9a46ead2592b3e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.180813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '540f94703ee9457bb9a46ead2592b3e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.181703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '540f94703ee9457bb9a46ead2592b3e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.191875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '540f94703ee9457bb9a46ead2592b3e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.198716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:41.220534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 880/1729] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:40 2022] DELETE /identity/v3/users/540f94703ee9457bb9a46ead2592b3e3 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:41.717107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:41.720150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:41.742156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070463fc-ec20-4a31-a76c-b9ac8a08168a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:41.869136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:41.869994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:41.870391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:41.870770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] PATH_INFO: `/v3/projects/c1137d4ebee94020aa5e0152a1b0205c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:41.872090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1137d4ebee94020aa5e0152a1b0205c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:41.874633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:41.913472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:41.914894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:41.918727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:41.954973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1137d4ebee94020aa5e0152a1b0205c', 'request_id': 'req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.963667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1137d4ebee94020aa5e0152a1b0205c', 'request_id': 'req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.964127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d672caa-ce99-4647-89c3-2617a187c582 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:41.970955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1137d4ebee94020aa5e0152a1b0205c', 'request_id': 'req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.991993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] The token cache is being invalidate because project c1137d4ebee94020aa5e0152a1b0205c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:41.992570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.993770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5b38ea6e-bef3-4eed-bdd8-ffe82436b023 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:41.997455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 850/1730] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:41 2022] DELETE /identity/v3/projects/c1137d4ebee94020aa5e0152a1b0205c => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:42.002762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9068e3-3848-4204-ac98-79b95d577c8d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.005865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9068e3-3848-4204-ac98-79b95d577c8d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.006187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9068e3-3848-4204-ac98-79b95d577c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.032119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-cb9068e3-3848-4204-ac98-79b95d577c8d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:42.033830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cb9068e3-3848-4204-ac98-79b95d577c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.035113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.035886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.036167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.036609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] PATH_INFO: `/v3/users/0fe86aa27b1741e38732e0e97834a4f5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.038096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fe86aa27b1741e38732e0e97834a4f5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.040517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.042553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 851/1731] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:42.052007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.052220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.052493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.073143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:42.074361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.083031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 852/1732] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:42.087699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.097336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8897d08f-eb0f-48f9-b177-c46d49f54149 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.102509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.102957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fe86aa27b1741e38732e0e97834a4f5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.103073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.103333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.103606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.104243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:42.104543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.106152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.112647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fe86aa27b1741e38732e0e97834a4f5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.113968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fe86aa27b1741e38732e0e97834a4f5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.114759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fe86aa27b1741e38732e0e97834a4f5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.119785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 853/1733] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:01:42.124143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.124361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94cb5be1-d560-41a7-acd4-5b6164aeb412 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fe86aa27b1741e38732e0e97834a4f5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.125824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.138919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14132158-277f-43c0-8c55-f9731730a622 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.145779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 881/1734] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:41 2022] DELETE /identity/v3/users/0fe86aa27b1741e38732e0e97834a4f5 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:42.149947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.150269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.150547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.155298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.156240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.156523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.156798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.157660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.160674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.170971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:42.172758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.196054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 854/1735] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/projects => generated 401 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:42.197047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.251959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 882/1736] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:42.262331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.263882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.271946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:42.276608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284ccfcf-3849-49a5-9ea6-46c062bf3996 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.282781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.283532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.283837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.284188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.285611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.288104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.313140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 855/1737] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/users => generated 301 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:42.314270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.325142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2ec35-188a-4fed-82d2-603167f85899 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.329838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.330491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.330775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.331051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.331746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.332513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f626495-1f8f-46a8-83d3-ae40fbbca478 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.333517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.343409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 856/1738] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:42.347148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.356888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094ed4bf-1000-4ec1-9a72-af75c45aa508 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.361643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.362251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.362532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.362809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.363992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.365543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.376833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 857/1739] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:42.380709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.382220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.382831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.383110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.383400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.390404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493d482d-0a4d-4ea0-88fb-b0d9c4331009 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.393299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.394950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd615f7c-b639-4534-b42c-88aba3a27859 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.395314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.395998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.396295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.396690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] PATH_INFO: `/v3/projects/7762e71669194f59a1e2cb5e948c5285/users/9d8f905e4fec4b05b6f5c982b3c40ceb/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.404912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 883/1740] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:42.414282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7762e71669194f59a1e2cb5e948c5285, user_id=9d8f905e4fec4b05b6f5c982b3c40ceb, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.416996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.439920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 858/1741] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:42 2022] PUT /identity/v3/projects/7762e71669194f59a1e2cb5e948c5285/users/9d8f905e4fec4b05b6f5c982b3c40ceb/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:42.441375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.441667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.441942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.473540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] MFA Rules not processed for user `9d8f905e4fec4b05b6f5c982b3c40ceb`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:42.474833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.542679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 884/1742] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:42.549715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.556313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=7,keystone:SELECT=87,keystone:INSERT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:42.615063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a387c813-fbf8-4757-9ebc-21bdafc4f65d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.620042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.620666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.620937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.621205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.621845 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.623783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.652480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 859/1743] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/projects => generated 397 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:42.656738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.671037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6ac7e-bb43-4b74-8ffa-5c63d3012ef5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.681254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.683443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.733912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:42.745950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.746598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.746883 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.747161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.748315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.750041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.778219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 885/1744] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/users => generated 305 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:42.779752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.790355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2740a355-9cbc-4bc1-baca-18f56e704145 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.798970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.799814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.799946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.800195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.800885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.802525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.819016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 886/1745] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:42.819230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081695a-48b9-400e-af77-6bfd9d0e5384 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.820419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.822538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.823295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.823913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.824290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] PATH_INFO: `/v3/projects/2adeeab71bd444d290f405908fb221fd` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.825502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2adeeab71bd444d290f405908fb221fd)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.827539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.831708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397db568-6af5-42b4-983d-dcc42c189567 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.838374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.839199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.839481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.839751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] PATH_INFO: `/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5/users/2faa9bcff254430a80daa728bda9e997/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.858863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1fb32aaf55c4f70aebfd21c31f41da5, user_id=2faa9bcff254430a80daa728bda9e997, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.861880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.877432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 887/1746] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:42 2022] PUT /identity/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5/users/2faa9bcff254430a80daa728bda9e997/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:42.881368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:42.886107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2adeeab71bd444d290f405908fb221fd', 'request_id': 'req-c3e76ec9-a053-4015-b98d-66507c873241'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.895175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2adeeab71bd444d290f405908fb221fd', 'request_id': 'req-c3e76ec9-a053-4015-b98d-66507c873241'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.903441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2adeeab71bd444d290f405908fb221fd', 'request_id': 'req-c3e76ec9-a053-4015-b98d-66507c873241'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.922527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] The token cache is being invalidate because project 2adeeab71bd444d290f405908fb221fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:42.922823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.923658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c3e76ec9-a053-4015-b98d-66507c873241 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:42.925364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 860/1747] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:42 2022] DELETE /identity/v3/projects/2adeeab71bd444d290f405908fb221fd => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:42.930909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff828495-3121-4739-995d-1c4e2c36e1b8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.931271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff828495-3121-4739-995d-1c4e2c36e1b8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.931548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff828495-3121-4739-995d-1c4e2c36e1b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.968711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ff828495-3121-4739-995d-1c4e2c36e1b8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:42.970199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ff828495-3121-4739-995d-1c4e2c36e1b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:42.973889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38165999-d361-4435-b8c3-285147fabff6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:42.979522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:42.986846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 861/1748] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:42.986999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.986999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.986999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.986999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:42.986999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:42.989879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:42.990282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:42.990655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:42.996822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 888/1749] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:01:43.000129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.001776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.014670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af9802-6c4a-4a93-b53c-ca1388b67707 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.018528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:43.020080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.020306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.020741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.021017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.021397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] PATH_INFO: `/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5/users/2faa9bcff254430a80daa728bda9e997/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.023665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1fb32aaf55c4f70aebfd21c31f41da5, user_id=2faa9bcff254430a80daa728bda9e997, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.026498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9600b9e1-bf9d-4f56-bac6-7b6219b942e9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.029178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 862/1750] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:43.033129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.051921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 889/1751] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:42 2022] PUT /identity/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5/users/2faa9bcff254430a80daa728bda9e997/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:43.052426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a48c7-bc7b-4db8-b898-b887aad5f544 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.052426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:43.053393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.053705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.053958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.083975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] MFA Rules not processed for user `2faa9bcff254430a80daa728bda9e997`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:43.086125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.115395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.116025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.116308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.116584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.117261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:43.117955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.119665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.132741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 863/1752] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:01:43.138226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.148943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f89e86-8499-4441-8f28-c443ecec37b3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.153587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.154216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.154502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.154778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.155400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.157424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.166298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 890/1753] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:43.177374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.178936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.195426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:43.198260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 864/1754] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/projects => generated 365 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:43.202490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.291328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e0a347-5839-437b-bb89-cb1c78c3e773 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.307319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f309d02-c7ef-472c-b7f0-43b0913f2f5f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.312167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.312785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.313067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.313343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.314001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.315972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.342521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 865/1755] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/users => generated 283 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:43.346985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.350640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.351391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.351746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.352117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.357560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24de99a-2256-424a-bd1e-058fc41782e9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.363850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.364101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.364388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.364663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.365418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.377475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.398931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 866/1756] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:43.399859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.415819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f51d0-f7b8-441d-b9a1-197b73ade6ae admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.420769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.421379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.421659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.421932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.422621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.424228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.435435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 867/1757] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:43.436437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.437944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c198cdab-3e45-404a-b330-229d6633e03d service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.439508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.448552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 891/1758] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:43.449709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f7b85-ab17-4f9b-9878-68a57fabb1b0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.454243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.454893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.455171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.455448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] PATH_INFO: `/v3/projects/6e627b1632544633ad0cd0725c0c3a1d/users/8c895a09139d4c3796c6b0bf49e66d22/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.471447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e627b1632544633ad0cd0725c0c3a1d, user_id=8c895a09139d4c3796c6b0bf49e66d22, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.476075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.513745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 868/1759] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:43 2022] PUT /identity/v3/projects/6e627b1632544633ad0cd0725c0c3a1d/users/8c895a09139d4c3796c6b0bf49e66d22/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:01:43.525729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.526025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.526366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.552778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.559289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] MFA Rules not processed for user `8c895a09139d4c3796c6b0bf49e66d22`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:43.565414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.591633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:43.642760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f8327-c807-40ca-8eec-d6e2bc42eb1e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.643920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 892/1760] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:01:43.647065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.648169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.648540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.648709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.648958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.651149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:43.659416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:43.739695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.739999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f93e74-b3ce-47bd-a719-4c3589a97ede None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.742045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.745348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.746003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.746296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.746570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.747264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.749280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.754699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 869/1761] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:43.801800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 893/1762] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/projects => generated 363 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:43.805853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.825278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a137bb9d-ab77-4c2c-9bc7-6c1ac6ec6441 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.914853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.917141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.917581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.918882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.921002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.950307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 870/1763] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:43 2022] POST /identity/v3/users => generated 288 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:43.956093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:43.967042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98e649-d5c4-4dce-8057-7e354dbe87ea admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:43.973786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:43.974581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:43.974961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:43.975310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:43.976252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:43.978379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:43.992699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 894/1764] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:43.997140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:44.013732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52026e3-38e7-49e6-9d32-c220598a49f9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:44.023859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:44.025028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.025809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.026389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] PATH_INFO: `/v3/projects/c5141b304e6b4cc7bb59f21d298f530b/users/37ccad83c1bd460e95f8437e7e1c4398/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.075583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5141b304e6b4cc7bb59f21d298f530b, user_id=37ccad83c1bd460e95f8437e7e1c4398, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:44.078447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:44.101108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 871/1765] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:43 2022] PUT /identity/v3/projects/c5141b304e6b4cc7bb59f21d298f530b/users/37ccad83c1bd460e95f8437e7e1c4398/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:44.104884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:44.118117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96dfbe-a1f1-473f-9d20-370cb4163a6b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:44.277649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:44.278582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.279834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.280991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.281741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:44.285995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b730967c-089f-4475-a412-f30c0aa23b9b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:44.314465 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 895/1766] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:44 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:44.315412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:44.328067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a74ae9-84f6-4cfe-af08-5f57d35c2ebd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:44.334260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:44.336069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.336361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.336677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] PATH_INFO: `/v3/projects/c5141b304e6b4cc7bb59f21d298f530b/users/37ccad83c1bd460e95f8437e7e1c4398/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.346050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5141b304e6b4cc7bb59f21d298f530b, user_id=37ccad83c1bd460e95f8437e7e1c4398, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:44.346050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:44.352381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 872/1767] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:44 2022] PUT /identity/v3/projects/c5141b304e6b4cc7bb59f21d298f530b/users/37ccad83c1bd460e95f8437e7e1c4398/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:44.362944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.363270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.363555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.391573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] MFA Rules not processed for user `37ccad83c1bd460e95f8437e7e1c4398`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:44.392828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:44.467658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 896/1768] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:44 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:44.479446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:44.481103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:44.513470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:44.541176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2d846-e482-43f0-a206-c293c502778b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:44.545033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:44.545634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.545907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.546209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.554922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:44.556337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:44.568301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 873/1769] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:44 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:44.670969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:44.680585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:44.685667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f859a4-4854-46fc-a454-560b304d3127 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:44.689563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:44.690232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:44.690518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:44.690794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:44.746773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:44.748220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f474d9-da55-43a5-8e46-b78f70b90688 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:44.757867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 897/1770] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:44 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:01:49.853161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:49.854642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:49.949122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b6f2e-650b-48b2-bd63-7e09d77c388e service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:49.956083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:49.957692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:49.958111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:49.958636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] PATH_INFO: `/v3/users/12e97df696b4444db047489b59e19feb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:49.959956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=12e97df696b4444db047489b59e19feb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:49.962648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:50.039946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12e97df696b4444db047489b59e19feb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.040711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12e97df696b4444db047489b59e19feb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.058494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12e97df696b4444db047489b59e19feb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.059370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12e97df696b4444db047489b59e19feb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.073706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12e97df696b4444db047489b59e19feb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.130437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 874/1771] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:49 2022] DELETE /identity/v3/users/12e97df696b4444db047489b59e19feb => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:50.135278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:50.136148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:50.136600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:50.182964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:50.184614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:50.298321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 898/1772] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:50.309363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:50.311156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:50.393780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbfac6f-e40d-4ddc-bfcf-458a39f60b8c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:50.397721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:50.398383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:50.398663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:50.398942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:50.405185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:50.410531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:50.412700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:50.424144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 875/1773] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:50.688511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:50.690102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:50.705040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33019361-1da7-4c59-baa5-be805fe3b8db None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:50.709809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:50.710462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:50.710747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:50.711017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] PATH_INFO: `/v3/projects/ee4896d989eb4479a566fab8448de5fd` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:50.712212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee4896d989eb4479a566fab8448de5fd)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:50.714110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:50.760595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee4896d989eb4479a566fab8448de5fd', 'request_id': 'req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.766019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee4896d989eb4479a566fab8448de5fd', 'request_id': 'req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.772325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee4896d989eb4479a566fab8448de5fd', 'request_id': 'req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.784294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] The token cache is being invalidate because project ee4896d989eb4479a566fab8448de5fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:50.784578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.785728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.787210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 899/1774] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:50 2022] DELETE /identity/v3/projects/ee4896d989eb4479a566fab8448de5fd => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:50.791173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:50.801534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4525dc7-5b42-4b75-9531-10153f709807 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:50.861459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:50.862220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:50.862467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:50.862775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] PATH_INFO: `/v3/users/d680df1fe14f42aebdd70a1c10f768bd` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:50.864001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] RBAC: Authorizing `identity:delete_user(user_id=d680df1fe14f42aebdd70a1c10f768bd)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:50.866050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:50.914825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd680df1fe14f42aebdd70a1c10f768bd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.915529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd680df1fe14f42aebdd70a1c10f768bd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.923226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd680df1fe14f42aebdd70a1c10f768bd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.923880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd680df1fe14f42aebdd70a1c10f768bd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.931425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-98d61787-486f-406f-8833-36df1e71c6df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd680df1fe14f42aebdd70a1c10f768bd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:50.947998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 876/1775] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:01:50 2022] DELETE /identity/v3/users/d680df1fe14f42aebdd70a1c10f768bd => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:51.241441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.242984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:51.261843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22ea78-5563-4dce-8bd2-e2bbea10cad4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.288061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:51.321543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.322225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.322509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.322781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] PATH_INFO: `/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.323916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0dbbe5d08fcf4739acc3c00d75b2c5b1)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.325812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.371244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0dbbe5d08fcf4739acc3c00d75b2c5b1', 'request_id': 'req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:51.377608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0dbbe5d08fcf4739acc3c00d75b2c5b1', 'request_id': 'req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:51.386786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0dbbe5d08fcf4739acc3c00d75b2c5b1', 'request_id': 'req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:51.404757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] The token cache is being invalidate because project 0dbbe5d08fcf4739acc3c00d75b2c5b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:01:51.405172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:51.405811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4fec7ffa-e5f8-4c51-b0ab-aaff487d0ed4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:01:51.407372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 900/1776] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:01:51 2022] DELETE /identity/v3/projects/0dbbe5d08fcf4739acc3c00d75b2c5b1 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:51.412534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.412819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.413090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.435313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:51.436663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:51.514041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 877/1777] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:51.516928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.517244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.517528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.539013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:51.540512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:51.548523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 901/1778] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:51.552585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.562235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e0920d-f73c-49d3-a729-412e53cb48d6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.567288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.567886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.568162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.568432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.569069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:01:51.569374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.571126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.579936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:51.586006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 878/1779] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:01:51.588655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.598252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71004e3-9393-4cb4-9733-5246c852f857 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.602946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.603864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.604130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.604409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.605048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.606956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.635544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 902/1780] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:51.639425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.649222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa865057-6afa-4044-800f-9879a8a0d2a8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.704524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.705129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.705423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.705691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.706409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.708374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.728262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 879/1781] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/users => generated 292 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:01:51.732320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.743159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195a12e-1871-4e41-892b-35ce8301d422 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.747624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.748219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.748494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.748761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.749394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.751006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.761155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 903/1782] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:51.764792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.774774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72334ae0-cd7d-4f6d-a89e-b7301a7aa2f5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.779512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.780155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.780428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.780696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.781341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.783001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83044c-cc32-4048-865c-45f4175ef152 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.794953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 880/1783] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:51.799225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:51.810246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d2d81b-aef4-43a9-bccc-cf6242ef0aaa admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:51.814948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:51.815616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.815926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.816762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] PATH_INFO: `/v3/projects/f7abd7242c834a6ba58e34ef57c4b12f/users/9226968546d54203b8649b6312f60805/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.832452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7abd7242c834a6ba58e34ef57c4b12f, user_id=9226968546d54203b8649b6312f60805, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:51.835168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:51.849779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 904/1784] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:51 2022] PUT /identity/v3/projects/f7abd7242c834a6ba58e34ef57c4b12f/users/9226968546d54203b8649b6312f60805/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:51.860772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:51.861057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:51.861329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:51.889361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] MFA Rules not processed for user `9226968546d54203b8649b6312f60805`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:51.890522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:51.952110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 881/1785] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:51.956556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.009908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:52.022181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0b511-aaae-403e-a3a6-00d878dfaa36 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.027423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.028240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.028526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.028803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.029534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.031452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.059068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 905/1786] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:01:51 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:52.062762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.073334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4e90bc-5273-41d7-bc43-7305a8dda3f9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.128175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.128915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.129196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.129790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.130122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.132027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.148241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:52.157637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 882/1787] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:01:52 2022] POST /identity/v3/users => generated 298 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:01:52.158798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.169889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7463e37-e46d-4d9b-8fc4-130c69e6c0f5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.174629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.175236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.176102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.176379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.177096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.178702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.189123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 906/1788] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:52.193349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.204056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59ea7d-edce-41cd-a4b0-222da3e419ca admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.210864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.211936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.212213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.212515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] PATH_INFO: `/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06/users/7aa8161ef9b5430a8377b3c4d115cbf6/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.230653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9b9bb3f75b2452182508d2bf0eb9a06, user_id=7aa8161ef9b5430a8377b3c4d115cbf6, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.233580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.251589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 883/1789] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:52 2022] PUT /identity/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06/users/7aa8161ef9b5430a8377b3c4d115cbf6/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:52.254957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.264849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af88bff-749c-43e1-9cc9-757868490579 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.325673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.326319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.326599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.326870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.327513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.329089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2ff94-b65c-4240-9103-f275cf5cac11 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 907/1790] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:01:52 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:01:52.343316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.353927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19474db3-fe64-4b27-a2aa-b77075a0ccb7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.358660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.359301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.359575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.359843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] PATH_INFO: `/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06/users/7aa8161ef9b5430a8377b3c4d115cbf6/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.361741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9b9bb3f75b2452182508d2bf0eb9a06, user_id=7aa8161ef9b5430a8377b3c4d115cbf6, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.364409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.374205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 884/1791] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:01:52 2022] PUT /identity/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06/users/7aa8161ef9b5430a8377b3c4d115cbf6/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:01:52.385307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.385667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.385950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.414454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] MFA Rules not processed for user `7aa8161ef9b5430a8377b3c4d115cbf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:52.415596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:52.482625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 908/1792] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:01:52 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:52.493430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.495267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:52.553544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce24d329-e53d-4e4f-858d-63a6f1d2c254 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.557182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.557819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.558144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.558420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.568359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.569957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb7f02c-dab2-4e81-9336-3ba9fe4526d1 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.580883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 885/1793] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:52 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:52.687561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:52.697888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d906af8f-f8e8-4310-92e4-3d334adfee27 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:52.701519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:52.702163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:52.702441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:52.702713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:52.750630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:01:52.763844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:52.765276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c90a6-baf8-4bb7-93b6-f7e54ebdb292 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:52.774888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 909/1794] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:01:57.315322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:57.315897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:57.315897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:57.317268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 886/1795] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:01:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:01:57.327208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:57.327681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:57.327915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:57.349376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:01:57.350621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:57.415553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 910/1796] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:01:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:01:57.427639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:01:57.429326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:01:57.446527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3da270-8031-45cd-bdaf-b522909b68ec None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:01:57.451890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:01:57.452506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:01:57.452787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:01:57.453064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:01:57.461632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:01:57.463237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22a0364-9a08-4da6-a001-e3d976f2cbdf service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:01:57.473892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 887/1797] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:01:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:01.736130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:01.738271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:01.803041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8878b02d-8b99-4c51-b1b3-8476254bf18c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:01.807740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:01.808413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:01.808690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:01.808990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] PATH_INFO: `/v3/users/91f37a04e7e44b8a8f79f23199b029aa` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:01.810159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91f37a04e7e44b8a8f79f23199b029aa)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:01.812230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:01.863492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91f37a04e7e44b8a8f79f23199b029aa'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:01.872828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91f37a04e7e44b8a8f79f23199b029aa'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:01.873641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91f37a04e7e44b8a8f79f23199b029aa'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:01.874521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91f37a04e7e44b8a8f79f23199b029aa'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:01.883849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91f37a04e7e44b8a8f79f23199b029aa'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:01.898372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:01.907278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 911/1798] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:01 2022] DELETE /identity/v3/users/91f37a04e7e44b8a8f79f23199b029aa => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:01.910648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:01.911323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:01.912647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:01.935781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:01.938068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:01.946246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:01.948149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.014332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 888/1799] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:02.022357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.023885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.033749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da37c82d-a1c7-4efd-b19f-0e7f4b546627 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.039383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.040028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.040306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.040581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] PATH_INFO: `/v3/users/9226968546d54203b8649b6312f60805` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.055224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9226968546d54203b8649b6312f60805)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.057492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.083729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8594ec8-3085-419a-96bc-30b70ad72171 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.087605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.088267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.088569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.088963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.096014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:02.102264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.103632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018dbf0-4a0c-4f0f-96a3-d5ef888f0fc6 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.116094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9226968546d54203b8649b6312f60805'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.122151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 889/1800] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:02.127104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9226968546d54203b8649b6312f60805'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.127958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9226968546d54203b8649b6312f60805'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.128693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9226968546d54203b8649b6312f60805'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.137938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9226968546d54203b8649b6312f60805'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.159252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 912/1801] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:01 2022] DELETE /identity/v3/users/9226968546d54203b8649b6312f60805 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:02.163965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.164256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.164556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.198173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:02.211183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.312885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 890/1802] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:02.328920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.333197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.409193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46f6129-9e9e-4595-a2f0-d62dd0ac8710 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.414429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.415184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.415473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.415799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.439523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.441151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.453259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 913/1803] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:02.547012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.550359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.571994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96be87c4-9d72-4ad4-859a-f4b66dd79584 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.583972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.585890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.586481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.588297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] PATH_INFO: `/v3/users/9d8f905e4fec4b05b6f5c982b3c40ceb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.591388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d8f905e4fec4b05b6f5c982b3c40ceb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.595732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.629627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.631285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.658448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d37749-84b0-4cb3-abee-3910d5f00d60 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.666876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.667721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.668163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.668448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] PATH_INFO: `/v3/projects/bd412080ef1e42b48b0014e3d1399915` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.669611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd412080ef1e42b48b0014e3d1399915)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.671914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.727351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d8f905e4fec4b05b6f5c982b3c40ceb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.728114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:02.734544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d8f905e4fec4b05b6f5c982b3c40ceb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.745713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d8f905e4fec4b05b6f5c982b3c40ceb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.746820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d8f905e4fec4b05b6f5c982b3c40ceb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.758593 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1b5c9daf-4c59-458b-ace6-5c45f525cb2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d8f905e4fec4b05b6f5c982b3c40ceb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.787509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 891/1804] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:02 2022] DELETE /identity/v3/users/9d8f905e4fec4b05b6f5c982b3c40ceb => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:02.791571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c709a87b-c191-472c-9f63-20c86a5075f1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.791929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c709a87b-c191-472c-9f63-20c86a5075f1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.792498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c709a87b-c191-472c-9f63-20c86a5075f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.816143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c709a87b-c191-472c-9f63-20c86a5075f1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:02.817704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c709a87b-c191-472c-9f63-20c86a5075f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.823405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd412080ef1e42b48b0014e3d1399915', 'request_id': 'req-648a2c75-86de-45fb-b6dd-0a203b66860a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.837842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd412080ef1e42b48b0014e3d1399915', 'request_id': 'req-648a2c75-86de-45fb-b6dd-0a203b66860a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.859008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd412080ef1e42b48b0014e3d1399915', 'request_id': 'req-648a2c75-86de-45fb-b6dd-0a203b66860a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.894201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] The token cache is being invalidate because project bd412080ef1e42b48b0014e3d1399915 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:02.894580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.895321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:02.902732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 914/1805] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:02 2022] DELETE /identity/v3/projects/bd412080ef1e42b48b0014e3d1399915 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:02.902732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.903268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 892/1806] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:02.903357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.903447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.903447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.903447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.918930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648a2c75-86de-45fb-b6dd-0a203b66860a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:02.925571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:02.926699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:02.927328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:02.927816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] PATH_INFO: `/v3/projects/f7abd7242c834a6ba58e34ef57c4b12f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:02.927986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:02.929227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7abd7242c834a6ba58e34ef57c4b12f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:02.931437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:02.935203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:02.937221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.948146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 893/1807] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:02.956858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:02.958765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:02.996714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7abd7242c834a6ba58e34ef57c4b12f', 'request_id': 'req-7fffc2fe-2727-4102-b825-00f72bdab9b1'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.002402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7abd7242c834a6ba58e34ef57c4b12f', 'request_id': 'req-7fffc2fe-2727-4102-b825-00f72bdab9b1'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.009662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7abd7242c834a6ba58e34ef57c4b12f', 'request_id': 'req-7fffc2fe-2727-4102-b825-00f72bdab9b1'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.019142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987a13e3-b030-4e58-83bb-0dcdb2847847 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.022338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] The token cache is being invalidate because project f7abd7242c834a6ba58e34ef57c4b12f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:03.022717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.023411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7fffc2fe-2727-4102-b825-00f72bdab9b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.025124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 915/1808] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:02 2022] DELETE /identity/v3/projects/f7abd7242c834a6ba58e34ef57c4b12f => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:03.028205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.028205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.028205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.046059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:03.047214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.071705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.072438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.072901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.073297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.075151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.117051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 916/1809] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:03.118412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.119705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.133151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f65a135-20d0-4bb1-b7bd-206c493d9f9c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.137827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.138501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.138783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.139058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] PATH_INFO: `/v3/users/7aa8161ef9b5430a8377b3c4d115cbf6` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.140194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=7aa8161ef9b5430a8377b3c4d115cbf6)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.142151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.148451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.149910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.162113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 894/1810] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:03.163256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.166916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:03.176243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ceffc-318e-4835-a172-00f6c96f06f9 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.182540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.183308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.183780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.184029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.184729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:03.185178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.186883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.202164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 895/1811] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:03.205704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.207743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7aa8161ef9b5430a8377b3c4d115cbf6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.217924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7aa8161ef9b5430a8377b3c4d115cbf6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.218787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7aa8161ef9b5430a8377b3c4d115cbf6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.219336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb63767a-f5ad-4d5e-8580-397e4f70542d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.219802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7aa8161ef9b5430a8377b3c4d115cbf6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.226088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.226767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.227094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.227376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.228537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.231602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7aa8161ef9b5430a8377b3c4d115cbf6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.232391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.267174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 917/1812] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:03 2022] DELETE /identity/v3/users/7aa8161ef9b5430a8377b3c4d115cbf6 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:03.280273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 896/1813] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:03 2022] POST /identity/v3/projects => generated 373 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:03.285085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.286719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.366299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807c77f2-02c2-4305-b464-bcdf679699bc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.431621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.432504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.432852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.433200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.434844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.437100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.462948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 918/1814] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:03 2022] POST /identity/v3/users => generated 287 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:03.467147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.484299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.484858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d4f48-4286-4922-884b-c72ffe1aa4b3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.487315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.499815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.500511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.500852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.501187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.501904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.502757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae08fe-fe1c-4132-94f6-8e10fdcfe665 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.505097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.508730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.509851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.510180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.510456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] PATH_INFO: `/v3/projects/7762e71669194f59a1e2cb5e948c5285` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.511746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7762e71669194f59a1e2cb5e948c5285)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.513751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.518483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 897/1815] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:03.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.535111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66da6c-8c1c-48a1-a7f0-19d302ee198c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.540634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.541245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.541521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.541808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.543951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.545544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.559958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 898/1816] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:03.564099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.569307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:03.576710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706c932c-817d-4f15-aed1-d58131b85470 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.584809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:03.594964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7762e71669194f59a1e2cb5e948c5285', 'request_id': 'req-3e499ea6-bc52-4cb9-9663-f01c0946a99b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.603700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7762e71669194f59a1e2cb5e948c5285', 'request_id': 'req-3e499ea6-bc52-4cb9-9663-f01c0946a99b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.614476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7762e71669194f59a1e2cb5e948c5285', 'request_id': 'req-3e499ea6-bc52-4cb9-9663-f01c0946a99b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.628326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] The token cache is being invalidate because project 7762e71669194f59a1e2cb5e948c5285 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:03.628617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.629299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.631089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 919/1817] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:03 2022] DELETE /identity/v3/projects/7762e71669194f59a1e2cb5e948c5285 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:03.631729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.633155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.651020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.651702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.651983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.652258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] PATH_INFO: `/v3/projects/74a16acf93654b72be2765fac8e3141f/users/cf13b506718c4b7bbd3fe3a2baf03ed2/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.668890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74a16acf93654b72be2765fac8e3141f, user_id=cf13b506718c4b7bbd3fe3a2baf03ed2, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.671602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.690251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 899/1818] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:03 2022] PUT /identity/v3/projects/74a16acf93654b72be2765fac8e3141f/users/cf13b506718c4b7bbd3fe3a2baf03ed2/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:03.691479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:03.693297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.723927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e499ea6-bc52-4cb9-9663-f01c0946a99b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.774018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350367e-b955-4a7e-b478-fc1653cef9b9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:03.779596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.780312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.780660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.781123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] PATH_INFO: `/v3/users/2faa9bcff254430a80daa728bda9e997` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.782458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2faa9bcff254430a80daa728bda9e997)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.782988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:03.783633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.783914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.784190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] PATH_INFO: `/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.784952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.785906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9b9bb3f75b2452182508d2bf0eb9a06)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:03.788463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:03.839862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2faa9bcff254430a80daa728bda9e997'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.840827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2faa9bcff254430a80daa728bda9e997'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.843593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9b9bb3f75b2452182508d2bf0eb9a06', 'request_id': 'req-2ce16382-19ac-4622-820d-6c2968448fcb'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.849940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2faa9bcff254430a80daa728bda9e997'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.851000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2faa9bcff254430a80daa728bda9e997'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.851597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9b9bb3f75b2452182508d2bf0eb9a06', 'request_id': 'req-2ce16382-19ac-4622-820d-6c2968448fcb'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.862148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa0e5371-07e5-4136-85ec-02398e79239f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2faa9bcff254430a80daa728bda9e997'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.864245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9b9bb3f75b2452182508d2bf0eb9a06', 'request_id': 'req-2ce16382-19ac-4622-820d-6c2968448fcb'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.873582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:03.881461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 900/1819] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:03 2022] DELETE /identity/v3/users/2faa9bcff254430a80daa728bda9e997 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:03.885903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.886382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.886723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.891835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] The token cache is being invalidate because project e9b9bb3f75b2452182508d2bf0eb9a06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:03.892212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.893417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2ce16382-19ac-4622-820d-6c2968448fcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:03.900540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 920/1820] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:03 2022] DELETE /identity/v3/projects/e9b9bb3f75b2452182508d2bf0eb9a06 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:03.903602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c8a49-8bd2-45c0-adaa-4767014e7da9 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:03.903941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c8a49-8bd2-45c0-adaa-4767014e7da9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:03.904222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c8a49-8bd2-45c0-adaa-4767014e7da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:03.910248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:03.920716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] MFA Rules not processed for user `cf13b506718c4b7bbd3fe3a2baf03ed2`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:03.922693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:03.930301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-961c8a49-8bd2-45c0-adaa-4767014e7da9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:03.932091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-961c8a49-8bd2-45c0-adaa-4767014e7da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.012649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 921/1821] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.027762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.028050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.028327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.032005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 901/1822] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:03 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.035338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.036854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.055869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8464983-852b-4deb-b768-138735ba90a0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.058439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:04.059659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.061024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.061890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.062260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.062529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.063496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.066230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.069505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 922/1823] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.088489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.096545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 902/1824] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:04.099794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe66487-ddee-47c9-abd0-0b2eebae84f4 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.100899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.176949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.177767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.178103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.178521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.179212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:04.179600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.181371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.183714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2e4f2-8830-4cc5-9c0d-0deeb20e5104 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.188408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.189021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.189298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.189567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.190222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.192869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.198408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 923/1825] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:02:04.202218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.214754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b9db3a-6253-42f7-8ff0-356fea604420 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.220015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.220635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.220913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.221182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.221838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.224057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.225537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 903/1826] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/users => generated 292 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:04.229412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.239894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa32818d-74dd-4e14-875a-bfdb511d6b73 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.245412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.251435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.251714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.251980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.252642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.254823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.261171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 924/1827] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/projects => generated 365 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:04.265031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.272309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 904/1828] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:04.272711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.296410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10ae1f8-8146-4822-bbdc-c56f81da1990 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.363815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9667bd-1aa7-484b-bcc7-a2fe474375d2 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.366223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:04.371019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.371019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.371276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.371540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.372180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.372914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.373775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.374129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.374598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.375049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] PATH_INFO: `/v3/projects/8808507d6ba949ccb3f08bd7f609d197/users/0da3efae62bb421287615c2a1207935c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.387113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:04.391397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8808507d6ba949ccb3f08bd7f609d197, user_id=0da3efae62bb421287615c2a1207935c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.394467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.407087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 925/1829] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/users => generated 283 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:04.407927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.409484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.414237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 905/1830] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:04 2022] PUT /identity/v3/projects/8808507d6ba949ccb3f08bd7f609d197/users/0da3efae62bb421287615c2a1207935c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:04.414237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.423799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4489215-7a7d-4f6b-ad9f-beb14ece11fb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.484177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.484819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.485097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.485370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] PATH_INFO: `/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.486236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e52388b-e4e6-47bc-a985-948128cdbe8c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.486503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1fb32aaf55c4f70aebfd21c31f41da5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.488926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.491378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.491985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.492320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.492599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.493237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.494881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.505388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 906/1831] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:04.506037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.515443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e79efb6-d010-43de-8ef5-168e9cade41b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.520134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.520720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.520995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.521270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.521895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.523504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.533439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 907/1832] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:04.533439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.540449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1fb32aaf55c4f70aebfd21c31f41da5', 'request_id': 'req-edb17811-cb32-403e-9f34-51ce101d9cfe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:04.543478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877e5c00-92e3-4fe8-8624-53bd68346b85 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.546781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1fb32aaf55c4f70aebfd21c31f41da5', 'request_id': 'req-edb17811-cb32-403e-9f34-51ce101d9cfe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:04.553995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1fb32aaf55c4f70aebfd21c31f41da5', 'request_id': 'req-edb17811-cb32-403e-9f34-51ce101d9cfe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:04.567133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] The token cache is being invalidate because project e1fb32aaf55c4f70aebfd21c31f41da5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:04.567419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:04.568050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:04.569789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 926/1833] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:04 2022] DELETE /identity/v3/projects/e1fb32aaf55c4f70aebfd21c31f41da5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:04.570512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.571808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.604308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.604941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.605218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.605490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.606196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.607804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17b91d7-e6bd-4475-ad4a-f342b85385bf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.623984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 908/1834] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:04.623984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.623984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.623984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.645232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:04.646537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.648343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb17811-cb32-403e-9f34-51ce101d9cfe admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.652877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.653548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.653816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.654123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] PATH_INFO: `/v3/projects/8808507d6ba949ccb3f08bd7f609d197/users/0da3efae62bb421287615c2a1207935c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.654707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 909/1835] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.655461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.656670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.661605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8808507d6ba949ccb3f08bd7f609d197, user_id=0da3efae62bb421287615c2a1207935c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.664299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1abc72c-b957-4c77-a87b-ac2e2e2f3141 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.668222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac99d4-4d79-4e28-8f90-2e7227bfa67e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.672909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.673561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.673843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.674154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] PATH_INFO: `/v3/projects/56c49780ee484f4fa9f8420f8de5c028/users/493043eb82704007ae6ad5ecb65feb44/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.674650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 927/1836] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:04 2022] PUT /identity/v3/projects/8808507d6ba949ccb3f08bd7f609d197/users/0da3efae62bb421287615c2a1207935c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:04.676482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f050-ea40-4635-846e-26c2dfa87a81 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.676769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f050-ea40-4635-846e-26c2dfa87a81 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.677045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7f050-ea40-4635-846e-26c2dfa87a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.691324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] RBAC: Authorizing `identity:create_grant(project_id=56c49780ee484f4fa9f8420f8de5c028, user_id=493043eb82704007ae6ad5ecb65feb44, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.693969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e67c8a0-c4b8-4d8b-a3fb-c017c489abdf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.696748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-37a7f050-ea40-4635-846e-26c2dfa87a81 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:04.698098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-37a7f050-ea40-4635-846e-26c2dfa87a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.707769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 910/1837] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:04 2022] PUT /identity/v3/projects/56c49780ee484f4fa9f8420f8de5c028/users/493043eb82704007ae6ad5ecb65feb44/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:04.708287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:04.709040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.709277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.709558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.742855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] MFA Rules not processed for user `0da3efae62bb421287615c2a1207935c`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:04.744131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.762156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:04.768821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 928/1838] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.770293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.770633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.770978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.799210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] MFA Rules not processed for user `493043eb82704007ae6ad5ecb65feb44`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:04.801005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.815959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 911/1839] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.816799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.866319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 929/1840] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:04.867409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.868712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:04.885755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7683e043-eb28-45a8-8caa-71db9e22ab91 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.890503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.891115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.891389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.891663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.892387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:04.892694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.894446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.907404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 912/1841] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:04.908827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.921060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a85b5-b376-4b45-8073-c577d6d6821e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.927488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.928129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.928405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.928689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.929321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.931243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.936790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431960f0-a5cf-443a-9c1b-cd614ccc95f6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:04.943168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:04.943878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:04.944222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:04.944562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:04.954460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:04.956001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:04.962699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 913/1842] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:04.963762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.970634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 930/1843] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:04 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:04.971555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:04.983944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7de912-2de7-4a4d-9bfd-5bd5ab17c47f service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.048442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.070165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29bf75-7d52-49f2-b937-02770f1e74c0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.073175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.073782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.074095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.074907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.075013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.076916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.077935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.078684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.079057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.079398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.080097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.082983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.107621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 931/1844] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/users => generated 290 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:05.108830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.111129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 914/1845] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:04 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:05.112566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.131261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.183201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238b84ea-86e2-41bb-b3a6-57ad4b1ff20d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.188752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575e3eec-e157-4fa2-b968-597330bf9458 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.195302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.196029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.196193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.196466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.197136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.198905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.209965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 915/1846] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:05.210852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.220424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65f6dd5-3d28-41d3-a99f-615ed706a888 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.226261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.226858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.227137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.227403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.228027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.229921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.241313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.242075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.242350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.242627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.244687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:05.254101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 916/1847] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:05 2022] POST /identity/v3/users => generated 291 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:05.258350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.270093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86496292-17c1-48e4-9f38-990760459510 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.276477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.277236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.277619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.277966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] PATH_INFO: `/v3/projects/016a750e99b94501af25c8f51c8ce14f/users/38d199d005ff41e398e9b2c5eb1d1bdb/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.294781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=016a750e99b94501af25c8f51c8ce14f, user_id=38d199d005ff41e398e9b2c5eb1d1bdb, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.298280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.315741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 917/1848] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:05 2022] PUT /identity/v3/projects/016a750e99b94501af25c8f51c8ce14f/users/38d199d005ff41e398e9b2c5eb1d1bdb/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:05.316771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.320133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.321571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.332353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 932/1849] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:05.333241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.344161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e1b070-c720-49d0-909f-c811a339c1db service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.398949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.403162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53bffcf-807f-4b9d-803c-d25569bba6c5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.410569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.411491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.411491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.411767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.412919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.414363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.414770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.415423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.415701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.415978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.416774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.418509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.426307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 918/1850] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:05.430176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.432750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 933/1851] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:05.438263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.447337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578d2cf0-edc2-453e-b8d6-b98a7ec76a8c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.448770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf9007-940a-4322-a6e3-26d19b74472b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.453367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.453700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.454404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.454629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.454739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.454950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] PATH_INFO: `/v3/projects/016a750e99b94501af25c8f51c8ce14f/users/38d199d005ff41e398e9b2c5eb1d1bdb/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.455192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.455468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.456633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.457580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] RBAC: Authorizing `identity:create_grant(project_id=016a750e99b94501af25c8f51c8ce14f, user_id=38d199d005ff41e398e9b2c5eb1d1bdb, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.458249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.460129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670f54d8-e15d-4c6f-b976-c59ae1858925 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.478754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 919/1853] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:05.482281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 934/1853] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:05 2022] PUT /identity/v3/projects/016a750e99b94501af25c8f51c8ce14f/users/38d199d005ff41e398e9b2c5eb1d1bdb/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:05.482842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.484154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.484305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.484586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.497345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff1ce2-b89e-4ca1-9a19-4ab288407464 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.521951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] MFA Rules not processed for user `38d199d005ff41e398e9b2c5eb1d1bdb`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:05.523380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:05.563937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.576302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.576302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.576601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.576850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] PATH_INFO: `/v3/projects/8bd1c6702e1a41adb7279caf4ff5b48d/users/1ae98f9d0d3b43698c50aa0116bf6ebc/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.592681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bd1c6702e1a41adb7279caf4ff5b48d, user_id=1ae98f9d0d3b43698c50aa0116bf6ebc, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.596197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b13aa-e94b-40df-8fce-76dbf9467aed admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.603941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 935/1854] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:05 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:05.613666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 920/1855] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:05 2022] PUT /identity/v3/projects/8bd1c6702e1a41adb7279caf4ff5b48d/users/1ae98f9d0d3b43698c50aa0116bf6ebc/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:05.620224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.625547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.625830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.626341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.663730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] MFA Rules not processed for user `1ae98f9d0d3b43698c50aa0116bf6ebc`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:05.665455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:05.708159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f4e0b-3547-4273-85ee-3e662ed50e44 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.732178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 921/1856] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:05.736907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.758195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.758820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.759103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.759379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.785744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.800038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42549ba-9eaf-476f-bb34-7eff4340903e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.805061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.805675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.805952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.806259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.806889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.808835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.821400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.822870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.834379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 936/1857] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:05.837765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 922/1858] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:05 2022] POST /identity/v3/projects => generated 381 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:05.841637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.852408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3780f8b-1c24-4584-b8c1-c975fe1417c8 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.911494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.912164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.912430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.912711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.913347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.915305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.919300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.936765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 937/1859] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:05 2022] POST /identity/v3/users => generated 297 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:05.941221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.950756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f244e-372e-4a0b-86d3-94b57bdc26c8 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.955354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.955940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.956256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.956534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.957173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:05.958914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:05.968791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 938/1860] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:05.973566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:05.976415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a29500c-9766-487d-a4a0-b58710d01e06 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.980853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.981500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.981832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.982212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:05.984125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd7b28-0a83-4688-ba0e-d31e2e0e80a7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:05.987977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:05.989491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:05.990110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:05.990396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:05.990779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] PATH_INFO: `/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f/users/c2cf13264df043de81e5abc484cd276d/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.006519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5da0916eeafd41a3a5303ac1fae27f3f, user_id=c2cf13264df043de81e5abc484cd276d, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.009238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:06.023529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 939/1861] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:05 2022] PUT /identity/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f/users/c2cf13264df043de81e5abc484cd276d/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:06.027587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:06.044024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.045480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:06.054891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 923/1862] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:05 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:06.093528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab65e5a-b8c4-411e-a659-91f533d2b96f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:06.099899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:06.100656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:06.100943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:06.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.101875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.103506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d163cc-1a54-485b-90b7-b79190bb276f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:06.118565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 940/1863] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:06.119733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:06.132905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd12935d-2540-4dfd-8bd3-8ad07f6e31e7 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:06.140528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:06.141442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:06.141779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:06.142160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] PATH_INFO: `/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f/users/c2cf13264df043de81e5abc484cd276d/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.144191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5da0916eeafd41a3a5303ac1fae27f3f, user_id=c2cf13264df043de81e5abc484cd276d, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.147067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:06.159977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 924/1864] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:06 2022] PUT /identity/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f/users/c2cf13264df043de81e5abc484cd276d/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:06.170743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:06.171039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:06.171321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.205969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:06.213188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] MFA Rules not processed for user `c2cf13264df043de81e5abc484cd276d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:06.215140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:06.322790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 941/1865] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:06 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:06.332367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:06.334435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:06.423401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4ba16-7f47-41da-b426-15bc39974963 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:06.428051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:06.428730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:06.428989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:06.429255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.438682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.440112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a6b443-3f7d-4d7b-a0e8-260f4e107cdb service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:06.449756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 925/1866] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:06.915877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:06.985812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa43dd9-e612-41d9-ba5a-42f1d138a03f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:06.990349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:06.990985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:06.991265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:06.991566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] PATH_INFO: `/v3/users/1ae98f9d0d3b43698c50aa0116bf6ebc` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:06.992786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ae98f9d0d3b43698c50aa0116bf6ebc)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:06.994763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:07.038683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ae98f9d0d3b43698c50aa0116bf6ebc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.045620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ae98f9d0d3b43698c50aa0116bf6ebc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.046269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ae98f9d0d3b43698c50aa0116bf6ebc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.046875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ae98f9d0d3b43698c50aa0116bf6ebc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.053788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ae98f9d0d3b43698c50aa0116bf6ebc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.071083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 942/1867] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:06 2022] DELETE /identity/v3/users/1ae98f9d0d3b43698c50aa0116bf6ebc => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:07.075593 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:07.075884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:07.076192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:07.095227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:07.096416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:07.156259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 926/1868] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:07.166726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:07.168162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:07.188197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:07.225904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e431dcaf-e6b8-4464-a2b6-63c5d39f14ac admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:07.229503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:07.230143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:07.230466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:07.230738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:07.240413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:07.242208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:07.252421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 943/1869] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:07.582734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:07.584254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:07.606949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4022e21e-5397-4ec9-9018-53a7e6594728 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:07.611965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:07.613113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:07.613385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:07.613657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] PATH_INFO: `/v3/projects/8bd1c6702e1a41adb7279caf4ff5b48d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:07.614811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bd1c6702e1a41adb7279caf4ff5b48d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:07.616715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:07.653403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:07.669381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bd1c6702e1a41adb7279caf4ff5b48d', 'request_id': 'req-4764338f-f331-489a-a486-2ffd030fd5fd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.675494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bd1c6702e1a41adb7279caf4ff5b48d', 'request_id': 'req-4764338f-f331-489a-a486-2ffd030fd5fd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.681596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bd1c6702e1a41adb7279caf4ff5b48d', 'request_id': 'req-4764338f-f331-489a-a486-2ffd030fd5fd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.699142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] The token cache is being invalidate because project 8bd1c6702e1a41adb7279caf4ff5b48d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:07.699439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.700301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.702603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 927/1870] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:07 2022] DELETE /identity/v3/projects/8bd1c6702e1a41adb7279caf4ff5b48d => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:07.706049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:07.717917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c52349-36c8-48ff-beef-ad4d7c649ebb service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:07.775714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:07.776346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:07.776622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:07.776891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] PATH_INFO: `/v3/users/c2cf13264df043de81e5abc484cd276d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:07.778008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2cf13264df043de81e5abc484cd276d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:07.779962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:07.825740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2cf13264df043de81e5abc484cd276d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.833060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2cf13264df043de81e5abc484cd276d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.833687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2cf13264df043de81e5abc484cd276d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.834463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2cf13264df043de81e5abc484cd276d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.841250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8bce39bd-8106-4196-aac5-d99f85f4a073 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2cf13264df043de81e5abc484cd276d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:07.858744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 944/1871] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:07 2022] DELETE /identity/v3/users/c2cf13264df043de81e5abc484cd276d => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:08.162141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.163573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:08.176765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4764338f-f331-489a-a486-2ffd030fd5fd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.236223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.236792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.237074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.237376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] PATH_INFO: `/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.238527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5da0916eeafd41a3a5303ac1fae27f3f)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.240502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.286631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5da0916eeafd41a3a5303ac1fae27f3f', 'request_id': 'req-1a711806-08cf-4eab-adfc-6ec8f9188250'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:08.292386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5da0916eeafd41a3a5303ac1fae27f3f', 'request_id': 'req-1a711806-08cf-4eab-adfc-6ec8f9188250'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:08.298528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5da0916eeafd41a3a5303ac1fae27f3f', 'request_id': 'req-1a711806-08cf-4eab-adfc-6ec8f9188250'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:08.310803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] The token cache is being invalidate because project 5da0916eeafd41a3a5303ac1fae27f3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:08.311092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:08.311746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1a711806-08cf-4eab-adfc-6ec8f9188250 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:08.313251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 928/1872] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:08 2022] DELETE /identity/v3/projects/5da0916eeafd41a3a5303ac1fae27f3f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:08.318167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.318633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.318739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.338525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:08.339727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:08.372741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:08.400691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 945/1873] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:08.408700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.408991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.409838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.430436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:08.431686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:08.439681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 929/1874] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:08.445898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.455773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e152c-2a0d-4c9c-b7c9-f7b96f7fba05 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.461031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.461643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.461926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.462426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.463083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:08.463396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.465394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.482020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 946/1875] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:08.484111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.493287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0533e7e-b41b-4719-81f5-5df3d3fb7c29 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.497891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.498523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.498803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.499083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.499714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.501630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.532134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 930/1876] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:08 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:08.536366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.554101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc0e8b-0ece-4c77-be48-34e193a84dda admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.617964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.618636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.618901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.619173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.619822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.621747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.646052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 947/1877] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:08 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:08.650049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.662521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b196-87b3-4522-9c4e-d00b5f0cf92b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.669561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.670400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.670863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.671130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.671771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.673431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.687344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 931/1878] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:08.688518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.699048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee594dc-3dfe-43c3-bb5e-c628011385d2 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.703767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.704393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.704670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.704961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.705582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.707252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279e20aa-3bcd-4142-a5da-438d8a27d72e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.718453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 948/1879] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:08.723235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:08.741839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea8fc5-63d5-4138-a474-ec3f29dbe020 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:08.767387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:08.768047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.768324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.769182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] PATH_INFO: `/v3/projects/8c62da947ba1444a84fb371c0b21fb61/users/b849e11c6f0148bab5d35d43980ebb06/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.773050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:08.793404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c62da947ba1444a84fb371c0b21fb61, user_id=b849e11c6f0148bab5d35d43980ebb06, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:08.796516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:08.822814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 932/1880] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:08 2022] PUT /identity/v3/projects/8c62da947ba1444a84fb371c0b21fb61/users/b849e11c6f0148bab5d35d43980ebb06/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:08.849012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9ccad-8cd0-478e-9f5c-7496c6d49596 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:08.849436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9ccad-8cd0-478e-9f5c-7496c6d49596 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:08.849800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9ccad-8cd0-478e-9f5c-7496c6d49596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:08.887337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7aa9ccad-8cd0-478e-9f5c-7496c6d49596 None None] MFA Rules not processed for user `b849e11c6f0148bab5d35d43980ebb06`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:08.889690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7aa9ccad-8cd0-478e-9f5c-7496c6d49596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:09.055489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 949/1881] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:09.071668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:09.073449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:09.144592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80087ced-73f7-499f-88a2-b4b28da2fa63 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:09.148436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:09.149060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:09.149336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:09.149645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:09.160561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:09.162143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4baf51-0272-4e3e-95d5-9b59e174d304 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:09.186337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 933/1882] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:15.461821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:15.462323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:15.462398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:15.463971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 950/1883] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:02:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:02:15.477544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:15.478183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:15.478272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:15.501765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:15.503008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:15.580637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 934/1884] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:02:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:15.594077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:15.595774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:15.614592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0cb86-fed7-4451-ae9f-ead25604261e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:15.620419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:15.621032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:15.621314 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:15.621589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:15.630859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:15.632515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:15.644531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 951/1885] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:16.414823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:16.416450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:16.568084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccb4f9-38d6-4338-8e22-6f40d2a34b13 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:16.579216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:16.579974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:16.580335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:16.580674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] PATH_INFO: `/v3/users/493043eb82704007ae6ad5ecb65feb44` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:16.582212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] RBAC: Authorizing `identity:delete_user(user_id=493043eb82704007ae6ad5ecb65feb44)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:16.586760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:16.637215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:16.686193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '493043eb82704007ae6ad5ecb65feb44'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.686869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '493043eb82704007ae6ad5ecb65feb44'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.696716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '493043eb82704007ae6ad5ecb65feb44'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.697431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '493043eb82704007ae6ad5ecb65feb44'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.712843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3ccbd5ab-d9e4-4203-9a1a-1e5f1161361e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '493043eb82704007ae6ad5ecb65feb44'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.725833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:16.727489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:16.754018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9e30c-8964-4b6f-affa-75067563399a service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:16.763099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:16.764205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:16.764663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:16.765126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] PATH_INFO: `/v3/users/8c895a09139d4c3796c6b0bf49e66d22` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:16.766596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c895a09139d4c3796c6b0bf49e66d22)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:16.768862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:16.773868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 935/1886] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:16 2022] DELETE /identity/v3/users/493043eb82704007ae6ad5ecb65feb44 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:16.785118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09adab6d-1085-4ae7-bb5d-0ec3e868ea43 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:16.785704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09adab6d-1085-4ae7-bb5d-0ec3e868ea43 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:16.786194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09adab6d-1085-4ae7-bb5d-0ec3e868ea43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:16.789919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:16.825374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-09adab6d-1085-4ae7-bb5d-0ec3e868ea43 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:16.827880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-09adab6d-1085-4ae7-bb5d-0ec3e868ea43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:16.843995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c895a09139d4c3796c6b0bf49e66d22'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.859175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c895a09139d4c3796c6b0bf49e66d22'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.859811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c895a09139d4c3796c6b0bf49e66d22'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.860852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c895a09139d4c3796c6b0bf49e66d22'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.869374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c895a09139d4c3796c6b0bf49e66d22'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:16.906404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 952/1887] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:16 2022] DELETE /identity/v3/users/8c895a09139d4c3796c6b0bf49e66d22 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:16.907801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:16.909598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:16.928492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 936/1888] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:16.930208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:16.930558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:16.930900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:16.961441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:16.963566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.016648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dc5f0d-8069-4a65-b6e8-6f9da8b12c44 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.025016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.025395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.025764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] PATH_INFO: `/v3/users/49e38ef4721d4011a2a82f84b0dcdb78` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.027245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=49e38ef4721d4011a2a82f84b0dcdb78)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.029681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:17.066194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 937/1889] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:17.066194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:17.077643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.117192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49e38ef4721d4011a2a82f84b0dcdb78'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:17.136080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49e38ef4721d4011a2a82f84b0dcdb78'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:17.137011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49e38ef4721d4011a2a82f84b0dcdb78'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:17.137941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49e38ef4721d4011a2a82f84b0dcdb78'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:17.157942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49e38ef4721d4011a2a82f84b0dcdb78'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:17.158532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:17.186293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 953/1890] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:16 2022] DELETE /identity/v3/users/49e38ef4721d4011a2a82f84b0dcdb78 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:17.186293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:17.187590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.195798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4ad263-5401-4863-ae44-58ac038e7776 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.208438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:17.254742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.255375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.255760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.256038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.258050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.266452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f8573-fcda-4b57-aa2a-726c6e440fbf admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.277398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.278204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.278553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.278897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.280982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.337250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.338932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1555d7-bd8c-4ea7-8d64-022575a14c2b service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:17.351791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 938/1891] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:17.353048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26229599-7e5b-40d3-a288-daf412350380 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.353419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26229599-7e5b-40d3-a288-daf412350380 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.353674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26229599-7e5b-40d3-a288-daf412350380 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.371709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.373222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:17.385432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 954/1892] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:17.388103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-26229599-7e5b-40d3-a288-daf412350380 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:17.389325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-26229599-7e5b-40d3-a288-daf412350380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.398087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 939/1893] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:17.408202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:17.409623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.421339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25454617-ad3d-4741-bb54-0dc50ac49ecc service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.425079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.425696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.425995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.426414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.437432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.439164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:17.450035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 955/1894] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:17.876025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26229599-7e5b-40d3-a288-daf412350380 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:17.879128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-26229599-7e5b-40d3-a288-daf412350380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.908171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:17.910192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:17.914868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26229599-7e5b-40d3-a288-daf412350380 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.921283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.923086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.923360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.924030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] PATH_INFO: `/v3/projects/56c49780ee484f4fa9f8420f8de5c028` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.925224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56c49780ee484f4fa9f8420f8de5c028)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.928122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:17.933649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a52ea-f688-4584-83a3-573a08743c54 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:17.947654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:17.948444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:17.948770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:17.949045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] PATH_INFO: `/v3/projects/33be387e591543919673f57dc809f643` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:17.950999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=33be387e591543919673f57dc809f643)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:17.963701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.021509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56c49780ee484f4fa9f8420f8de5c028', 'request_id': 'req-1d20cc42-c5dc-481a-877a-14c6633f8e03'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.023508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33be387e591543919673f57dc809f643', 'request_id': 'req-8740f190-54f2-49dc-b069-aaa4c38224cc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.027410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56c49780ee484f4fa9f8420f8de5c028', 'request_id': 'req-1d20cc42-c5dc-481a-877a-14c6633f8e03'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.029771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33be387e591543919673f57dc809f643', 'request_id': 'req-8740f190-54f2-49dc-b069-aaa4c38224cc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.034583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56c49780ee484f4fa9f8420f8de5c028', 'request_id': 'req-1d20cc42-c5dc-481a-877a-14c6633f8e03'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.036481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33be387e591543919673f57dc809f643', 'request_id': 'req-8740f190-54f2-49dc-b069-aaa4c38224cc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.048381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] The token cache is being invalidate because project 56c49780ee484f4fa9f8420f8de5c028 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:18.048675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.049424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.050908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] The token cache is being invalidate because project 33be387e591543919673f57dc809f643 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:18.051031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 940/1895] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:17 2022] DELETE /identity/v3/projects/56c49780ee484f4fa9f8420f8de5c028 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.054530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.054639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.054724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.054818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.054909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 956/1896] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:17 2022] DELETE /identity/v3/projects/33be387e591543919673f57dc809f643 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.055372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.056713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.082713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:18.094157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:18.124234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20cc42-c5dc-481a-877a-14c6633f8e03 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.125011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8740f190-54f2-49dc-b069-aaa4c38224cc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.129103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.129639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.129776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.130016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.130292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] PATH_INFO: `/v3/projects/6e627b1632544633ad0cd0725c0c3a1d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.130392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.130572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.131410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] PATH_INFO: `/v3/users/b849e11c6f0148bab5d35d43980ebb06` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.132052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e627b1632544633ad0cd0725c0c3a1d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.132544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b849e11c6f0148bab5d35d43980ebb06)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.133962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.134510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.183796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e627b1632544633ad0cd0725c0c3a1d', 'request_id': 'req-db373e2b-11aa-4487-afee-1951bd8fbb71'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.190549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e627b1632544633ad0cd0725c0c3a1d', 'request_id': 'req-db373e2b-11aa-4487-afee-1951bd8fbb71'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.193726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b849e11c6f0148bab5d35d43980ebb06'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.198613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e627b1632544633ad0cd0725c0c3a1d', 'request_id': 'req-db373e2b-11aa-4487-afee-1951bd8fbb71'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.204540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b849e11c6f0148bab5d35d43980ebb06'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.205222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b849e11c6f0148bab5d35d43980ebb06'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.205892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b849e11c6f0148bab5d35d43980ebb06'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.216015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b849e11c6f0148bab5d35d43980ebb06'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.216478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] The token cache is being invalidate because project 6e627b1632544633ad0cd0725c0c3a1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:18.216833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.217724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.219609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 941/1897] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:18 2022] DELETE /identity/v3/projects/6e627b1632544633ad0cd0725c0c3a1d => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.220520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.235033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.254276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 957/1898] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:18 2022] DELETE /identity/v3/users/b849e11c6f0148bab5d35d43980ebb06 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.254276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.254276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.326543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46271854-cade-4c47-8725-5df4ffdc360f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.326780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db373e2b-11aa-4487-afee-1951bd8fbb71 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.332413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.332586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.333058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.333177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.333342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.333444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.333670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] PATH_INFO: `/v3/users/38d199d005ff41e398e9b2c5eb1d1bdb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.333849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] PATH_INFO: `/v3/users/44410a2bf56548c3ba5884f436925d7a` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.335294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44410a2bf56548c3ba5884f436925d7a)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.335650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] RBAC: Authorizing `identity:delete_user(user_id=38d199d005ff41e398e9b2c5eb1d1bdb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.337278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.337689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.404904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44410a2bf56548c3ba5884f436925d7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.410342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38d199d005ff41e398e9b2c5eb1d1bdb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.411043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38d199d005ff41e398e9b2c5eb1d1bdb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.416000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44410a2bf56548c3ba5884f436925d7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.416764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44410a2bf56548c3ba5884f436925d7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.417496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44410a2bf56548c3ba5884f436925d7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.419052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:18.421246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38d199d005ff41e398e9b2c5eb1d1bdb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.422230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38d199d005ff41e398e9b2c5eb1d1bdb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.423363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:18.431404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38d199d005ff41e398e9b2c5eb1d1bdb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.433284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7a7b2472-ee3d-41a0-8dd1-b08ba0ee1660 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44410a2bf56548c3ba5884f436925d7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.452235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 942/1899] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:18 2022] DELETE /identity/v3/users/38d199d005ff41e398e9b2c5eb1d1bdb => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.455752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 958/1900] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:18 2022] DELETE /identity/v3/users/44410a2bf56548c3ba5884f436925d7a => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:18.456088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.459575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.467381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.467671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.467953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.489492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:18.491528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.570280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 959/1901] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:18.601027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:18.603623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:18.614938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601793b0-0b5a-4e08-b975-fe788e29091d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.622294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.624591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.625185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.626992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] PATH_INFO: `/v3/users/37ccad83c1bd460e95f8437e7e1c4398` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.630454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=37ccad83c1bd460e95f8437e7e1c4398)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.632668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.703477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa755cf-ffc6-46da-a695-82312144c7fd None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:18.708503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:18.709795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:18.710108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:18.710387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:18.728286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:18.730377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:18.736532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37ccad83c1bd460e95f8437e7e1c4398'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.737318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37ccad83c1bd460e95f8437e7e1c4398'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.751311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 960/1902] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:18.756777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37ccad83c1bd460e95f8437e7e1c4398'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.757558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37ccad83c1bd460e95f8437e7e1c4398'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.768423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37ccad83c1bd460e95f8437e7e1c4398'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:18.790954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 943/1903] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:18 2022] DELETE /identity/v3/users/37ccad83c1bd460e95f8437e7e1c4398 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:19.092643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:19.094944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.113409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56d107a-9f16-4acf-b50b-caaf74f8331f service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:19.219601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:19.220276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.220578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.220875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] PATH_INFO: `/v3/projects/54ffd40154a249b892dfcf49adffa6c1` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.222335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] RBAC: Authorizing `identity:delete_project(project_id=54ffd40154a249b892dfcf49adffa6c1)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:19.225330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:19.294838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:19.296285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.307201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:19.317513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0401cf-b4eb-4307-ae57-132754c976fb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:19.324453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:19.325157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.325450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.325830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] PATH_INFO: `/v3/projects/8c62da947ba1444a84fb371c0b21fb61` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.328695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c62da947ba1444a84fb371c0b21fb61)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:19.330794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:19.361818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54ffd40154a249b892dfcf49adffa6c1', 'request_id': 'req-8947d883-a573-48a5-a780-98b881050bbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.377088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54ffd40154a249b892dfcf49adffa6c1', 'request_id': 'req-8947d883-a573-48a5-a780-98b881050bbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.386924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54ffd40154a249b892dfcf49adffa6c1', 'request_id': 'req-8947d883-a573-48a5-a780-98b881050bbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.417855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] The token cache is being invalidate because project 54ffd40154a249b892dfcf49adffa6c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:19.418236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.419158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.428031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 961/1904] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:19 2022] DELETE /identity/v3/projects/54ffd40154a249b892dfcf49adffa6c1 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:19.431373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:19.434390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.438338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:19.442908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c62da947ba1444a84fb371c0b21fb61', 'request_id': 'req-356930c6-f40f-4004-9b9a-fc20d1631d21'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.453627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c62da947ba1444a84fb371c0b21fb61', 'request_id': 'req-356930c6-f40f-4004-9b9a-fc20d1631d21'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.467884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c62da947ba1444a84fb371c0b21fb61', 'request_id': 'req-356930c6-f40f-4004-9b9a-fc20d1631d21'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.500998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] The token cache is being invalidate because project 8c62da947ba1444a84fb371c0b21fb61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:19.501817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.502825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-356930c6-f40f-4004-9b9a-fc20d1631d21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.505176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 944/1905] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:19 2022] DELETE /identity/v3/projects/8c62da947ba1444a84fb371c0b21fb61 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:19.507386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6831d4-f0ae-4e87-ac36-abb46908bce1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.509055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6831d4-f0ae-4e87-ac36-abb46908bce1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.509652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6831d4-f0ae-4e87-ac36-abb46908bce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.548151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-fd6831d4-f0ae-4e87-ac36-abb46908bce1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:19.549006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8947d883-a573-48a5-a780-98b881050bbc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:19.549383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fd6831d4-f0ae-4e87-ac36-abb46908bce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.633154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:19.633738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.634524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.634826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] PATH_INFO: `/v3/projects/016a750e99b94501af25c8f51c8ce14f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.636425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] RBAC: Authorizing `identity:delete_project(project_id=016a750e99b94501af25c8f51c8ce14f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:19.638740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:19.647453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 945/1906] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:19.648990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.649566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.649706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.675201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:19.676417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.692154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 946/1907] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:19.692154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:19.692154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.703624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d8ba21-96f3-4354-96ee-c29fbdb2c807 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:19.712913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '016a750e99b94501af25c8f51c8ce14f', 'request_id': 'req-caa6b342-529d-421a-8cbe-64c983646fbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.722303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:19.726494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '016a750e99b94501af25c8f51c8ce14f', 'request_id': 'req-caa6b342-529d-421a-8cbe-64c983646fbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.739533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '016a750e99b94501af25c8f51c8ce14f', 'request_id': 'req-caa6b342-529d-421a-8cbe-64c983646fbc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.754772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] The token cache is being invalidate because project 016a750e99b94501af25c8f51c8ce14f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:19.755255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.756435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caa6b342-529d-421a-8cbe-64c983646fbc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.762145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 962/1908] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:19 2022] DELETE /identity/v3/projects/016a750e99b94501af25c8f51c8ce14f => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:19.762145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f77dc-b0b9-4514-b02f-8295a7f19a43 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.762145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f77dc-b0b9-4514-b02f-8295a7f19a43 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.762145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5f77dc-b0b9-4514-b02f-8295a7f19a43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.781357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ac5f77dc-b0b9-4514-b02f-8295a7f19a43 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:19.783066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ac5f77dc-b0b9-4514-b02f-8295a7f19a43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.791822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:19.792733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.793064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.793678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] PATH_INFO: `/v3/projects/c5141b304e6b4cc7bb59f21d298f530b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.794917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5141b304e6b4cc7bb59f21d298f530b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:19.797640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:19.848927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 963/1909] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:19.851634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.852084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.852383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.866453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5141b304e6b4cc7bb59f21d298f530b', 'request_id': 'req-e327522b-a4d7-4b2f-9756-b0e575dd7968'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.882932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5141b304e6b4cc7bb59f21d298f530b', 'request_id': 'req-e327522b-a4d7-4b2f-9756-b0e575dd7968'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.883553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:19.889651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5141b304e6b4cc7bb59f21d298f530b', 'request_id': 'req-e327522b-a4d7-4b2f-9756-b0e575dd7968'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.897008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:19.902676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:19.913342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] The token cache is being invalidate because project c5141b304e6b4cc7bb59f21d298f530b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:19.913493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.915535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e327522b-a4d7-4b2f-9756-b0e575dd7968 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:19.920328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 947/1910] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:19 2022] DELETE /identity/v3/projects/c5141b304e6b4cc7bb59f21d298f530b => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:19.925735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:19.926008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:19.929132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:19.955909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:19.957175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.010877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 964/1911] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:20.015004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.015004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.026365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5f0775-45e5-429a-a7c0-d373eb7169d1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.032874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.033572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.033836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.034264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.035100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:20.035359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.037651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453a801-b778-4c1d-928d-e79b55fccb80 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.057296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 965/1912] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:20.057296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.057296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.057296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.059530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 948/1913] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:20.060328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.077771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69ccc73-360c-41bd-8de9-175d88019792 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.078432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:20.079659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.085924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.086757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.087794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.088141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.088898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:20.089314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.091051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90137677-e042-47f7-90b7-0080d7de62a4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.095046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 966/1914] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:20.095748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.107849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 949/1915] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:20.109689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524df4a9-ef4e-473e-9321-55161a3d8c5c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.115394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa574a8-ff0a-4d26-9efc-9b7e68ad41f3 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.120069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524df4a9-ef4e-473e-9321-55161a3d8c5c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.120401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524df4a9-ef4e-473e-9321-55161a3d8c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.123598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.124739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.125331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.125848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.126818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.129427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.145785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-524df4a9-ef4e-473e-9321-55161a3d8c5c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:20.147304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-524df4a9-ef4e-473e-9321-55161a3d8c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.156593 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 950/1916] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:20.160984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.161277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.161562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.164850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 967/1917] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/projects => generated 407 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.166003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.188053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:20.189327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.225216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:20.247045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612022cb-3ad0-4a98-b74b-ab7644374525 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.251964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.252614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.252890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.253155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.253782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.255772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.256846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 951/1918] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:20.258907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.273705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6561481-8cd9-4402-bb31-910fad3b78a2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.279716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.280150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.280427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.280695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.281663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:20.281663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.283399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.288174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 968/1919] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/projects => generated 381 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.289144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.301398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 952/1920] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:20.302345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.314693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c7d30-9ad3-43c6-8e8a-f0a1311fcbd7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.365096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f07f2ad-c4f6-4c07-8a30-d6f246da8573 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.371454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.372149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.372497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.372785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.373480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.374978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.375442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.375659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.376075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.376357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.377007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:20.377315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.378932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.389435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:20.394910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 953/1921] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:20.395750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.402598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 969/1922] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/users => generated 304 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.404602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.407273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81b259b-e591-4484-a1ff-2106aeb8fe4d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.412647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.413346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.413592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.413871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.414539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.416794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.417617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9352b06-eab5-4802-ab75-a3b283bdbe93 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.422228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.422833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.423110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.423380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.424317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.425879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.442248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 954/1923] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/users => generated 291 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.442248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.451701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b96fd-1744-4929-a0f6-73e402899d6a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.456814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.457419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.457705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.458000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.458648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.461113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.462082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 970/1924] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/projects => generated 359 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.465176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.494400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 955/1925] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/projects => generated 387 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.494400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.531710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a2d21c-db05-4002-8f18-459025313721 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.564130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9631e6-851e-49cd-99fb-e1f812b3fe65 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.569454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.570097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.570377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.570654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.571312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.572902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.578123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:20.586361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 956/1926] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.587042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.599914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.600540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.601277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.601277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.601710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.603388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.604634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba89ce4-b659-452b-89ec-86f25fd6bdf6 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.611615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.612744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.613031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.613309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.613950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.618392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 971/1927] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.618392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.618604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.627540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3801b896-49e6-4fed-8bd0-8b05b600aa63 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.633411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.634222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.634559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.634873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.635678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.637910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.644017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 957/1928] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/users => generated 280 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.645029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.657783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1023fec-2109-4a6d-8250-9bdd2ca05d13 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.664682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.665566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.665916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.666222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.666924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.670920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 972/1929] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/users => generated 294 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:20.671050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.671147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.686887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22cf15c-1485-4d1e-8656-0252241f048f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.692293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 958/1930] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.692293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.695735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.696372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.696650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.696923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.697560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.699274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.699935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e07411-d139-4ab7-9adc-3980425d1569 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.704550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.705139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.705417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.705696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.706359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.707922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.711318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 973/1931] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.712258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.719682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 959/1932] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.720340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.721456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44373130-839b-40c6-b65c-0ae71fa8330f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.726250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.726840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.727115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.727388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.728015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.729575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.729751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b97c90d-91d2-4472-b774-3097986a2929 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.734864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.735718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.735999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.736274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] PATH_INFO: `/v3/projects/09ea94dc884f483dab8cc80ccdf5b935/users/31d292b6801f433e844182e8e4501960/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.740194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 974/1933] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.742684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.753043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39307ef1-5315-4162-b5a0-54a5ee5d2e15 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.756346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=09ea94dc884f483dab8cc80ccdf5b935, user_id=31d292b6801f433e844182e8e4501960, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.758766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.759433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.759713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.759985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] PATH_INFO: `/v3/projects/3a37a636f0a3487b9978e430ffa96da8/users/eac48c4521b0489a8530766ddc01a0ef/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.760867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.775853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 960/1934] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:20 2022] PUT /identity/v3/projects/09ea94dc884f483dab8cc80ccdf5b935/users/31d292b6801f433e844182e8e4501960/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:20.776052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a37a636f0a3487b9978e430ffa96da8, user_id=eac48c4521b0489a8530766ddc01a0ef, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.778592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.780094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.792861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 975/1935] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:20 2022] PUT /identity/v3/projects/3a37a636f0a3487b9978e430ffa96da8/users/eac48c4521b0489a8530766ddc01a0ef/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:20.793848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:20.846611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:20.860070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf9b35-a70c-439b-b9a1-e16900146a6a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.874085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ff1c8e-9a59-4a4b-b688-172e7995d1a8 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:20.879219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.879846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.880128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.880405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.881638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.883401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf44d4b1-c891-4757-b2fe-c06ccdf4f05b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.894352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 976/1936] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.895618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.896054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.896451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.930272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] MFA Rules not processed for user `31d292b6801f433e844182e8e4501960`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:20.931713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.938001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:20.938699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.938979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.939258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.940089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:20.942180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0940edc1-5b01-4c4f-811d-8651238211d6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:20.952827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 961/1937] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:20.954064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:20.954353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:20.954627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:20.981202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] MFA Rules not processed for user `eac48c4521b0489a8530766ddc01a0ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:20.982461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:20.996528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:21.000282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 977/1938] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:21.004485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.015173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bdf971-5ede-4b74-b06b-516d41c29106 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.020437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.021110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.021417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.021692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] PATH_INFO: `/v3/projects/d3f57fcae0634fdfa444e248abde5ee5/users/8c16c13f025b4f0898c184904db97a28/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.051278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3f57fcae0634fdfa444e248abde5ee5, user_id=8c16c13f025b4f0898c184904db97a28, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.054916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.059970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 962/1939] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:20 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:21.062932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.075025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 978/1940] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:21 2022] PUT /identity/v3/projects/d3f57fcae0634fdfa444e248abde5ee5/users/8c16c13f025b4f0898c184904db97a28/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:21.075927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc29826d-d40c-4593-9704-915ddd85e4e8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.076091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.164932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f397a179-81de-481c-a9d1-9c4a7319d136 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.170219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.170415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.170914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.171128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.171238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.171457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.171701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.172193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.173112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] PATH_INFO: `/v3/projects/4c22e5faa8494d349aa679fd4e219399/users/f7d19108cc5d4352a060eb053ab7f1e5/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.174263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.196839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c22e5faa8494d349aa679fd4e219399, user_id=f7d19108cc5d4352a060eb053ab7f1e5, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.200632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4247d-c323-4ce7-8607-d231e56b0755 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.207917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 979/1941] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/projects => generated 385 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.209038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.224216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 963/1942] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:21 2022] PUT /identity/v3/projects/4c22e5faa8494d349aa679fd4e219399/users/f7d19108cc5d4352a060eb053ab7f1e5/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:21.239235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.239537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.239792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.269418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:21.290094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] MFA Rules not processed for user `8c16c13f025b4f0898c184904db97a28`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:21.291518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:21.324506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb9e641-8087-4ee9-9891-ef66f6dda542 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.399521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 964/1943] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:21.400582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.405831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.406620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.406904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.407181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.407825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.409878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aec705-8804-46e8-b42b-deaaa7d008cd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.424833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:21.438878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 980/1944] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/projects => generated 409 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.440102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.440382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.440656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.468574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed74e6c7-3888-4b2f-9970-c082d882562e None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.468797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] MFA Rules not processed for user `f7d19108cc5d4352a060eb053ab7f1e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:21.470099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:21.539399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 981/1945] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:21.540629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.544135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.544776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.545056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.545330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.546569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.548062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.553604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e467a6d-1957-4c61-8bd6-9170477ba2e0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.558652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.559424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.559733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.560004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.561175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.564440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.584865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 965/1946] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/users => generated 293 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.584865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.594779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668d53ed-4003-446e-9235-3f0e29c91e89 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.600442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.601117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.604432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 982/1947] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/projects => generated 389 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.604582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.604582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.604761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.604847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.604847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.628364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 966/1948] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/users => generated 311 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.632876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.634986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:21.657347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:21.671646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1a0ad6-8daa-45c2-929b-94ef535a989a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.676426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.677270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.678138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.678790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.680385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.697615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 983/1949] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:21.699283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.703242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81e17dc-25af-4047-b65c-b4c458430363 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.712320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ebb32-291b-4a63-94d3-a76a973ca390 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.718620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.719369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.719651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.719923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.720630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.722858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.757539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 984/1950] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:21 2022] POST /identity/v3/users => generated 301 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:21.758642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.772898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f20c13-4c58-476e-b714-4b3b1a67aaca admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.775647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.776454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.776600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.776881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.779685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.780575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.780958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.781329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.782113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.783872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.796329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 985/1951] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:21.797315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.808447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d7e047-4f00-4f68-b294-0b7a306e3400 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.813845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.814731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.815112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.815486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.816471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.818941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.833392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 986/1952] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:21.834455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.846565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9cc50-9510-4603-b47a-24859fffbd41 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.851810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.852496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.852864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.853202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.854175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.856036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.857779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.859862 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.867413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 987/1953] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:21.868279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.875143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 967/1954] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:21 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:21.877823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.878166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d87c0f-4934-4f4b-b189-5e73901229db admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.884081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.884756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.885032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.885301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] PATH_INFO: `/v3/projects/4d14d4209d9d4e7790e362d747e56dd7/users/a89740eb26a947e98eeb358179464e32/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.887748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:21.898094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d7172-283d-4fb9-9aa5-54738f3ad5fb service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:21.902784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:21.903442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:21.903722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:21.903998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] PATH_INFO: `/v3/projects/cf6327e570ee4dbb83b659ec54da55fb/users/eb58a5e0d4654b8797f7e7cb96ed9c46/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:21.904395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d14d4209d9d4e7790e362d747e56dd7, user_id=a89740eb26a947e98eeb358179464e32, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.908166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.919435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf6327e570ee4dbb83b659ec54da55fb, user_id=eb58a5e0d4654b8797f7e7cb96ed9c46, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:21.921453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 988/1955] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:21 2022] PUT /identity/v3/projects/4d14d4209d9d4e7790e362d747e56dd7/users/a89740eb26a947e98eeb358179464e32/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:21.922108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:21.922252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.936234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 968/1956] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:21 2022] PUT /identity/v3/projects/cf6327e570ee4dbb83b659ec54da55fb/users/eb58a5e0d4654b8797f7e7cb96ed9c46/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:21.937341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:21.989054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726fec4-1156-4cea-93a6-822fb0c76b4e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.019371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77b034-ed01-4bec-8927-5b04e7e40028 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.024637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.025475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.025818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.026148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] PATH_INFO: `/v3/users/f7d19108cc5d4352a060eb053ab7f1e5` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.027327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7d19108cc5d4352a060eb053ab7f1e5)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.029740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.035169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:22.061109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.061868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.062188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.062481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] PATH_INFO: `/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904/users/1314e33fd81647cfa5537bd2bcd37e4c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.080928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba2cbb69e7644d9b0f6d75f86b06904, user_id=1314e33fd81647cfa5537bd2bcd37e4c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.084504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.087422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7d19108cc5d4352a060eb053ab7f1e5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:22.088070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7d19108cc5d4352a060eb053ab7f1e5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:22.097864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7d19108cc5d4352a060eb053ab7f1e5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:22.098801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7d19108cc5d4352a060eb053ab7f1e5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:22.113914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 989/1957] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:21 2022] PUT /identity/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904/users/1314e33fd81647cfa5537bd2bcd37e4c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:22.114876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.119162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.119290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22564230-3a8a-4767-9d93-9d9ce56b6307 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7d19108cc5d4352a060eb053ab7f1e5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:22.139474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 969/1958] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:21 2022] DELETE /identity/v3/users/f7d19108cc5d4352a060eb053ab7f1e5 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:22.142629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.142629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.142629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.169151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] MFA Rules not processed for user `eb58a5e0d4654b8797f7e7cb96ed9c46`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:22.170409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.187276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af886f04-8d0f-47a4-9e7c-a7bb73b28035 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.206328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:22.237012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 970/1959] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:22.237861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.239144 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.256698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.257323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.257605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.257881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.258562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.260172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5e674e-d4f8-43a0-983b-17536881bc44 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.270286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 990/1960] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:22.272207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.272638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.272900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.292333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:22.293538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.301141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 991/1961] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:22.301988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.303281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.312883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a795dd10-f6a7-449a-b402-c0fe02bcfe20 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.314816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882faac-60c8-4741-aacf-b70ae6422563 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.317557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.318226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.318504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.318772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.319407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.319522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.320090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.320372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.320646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.320988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.321271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.323194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.331521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 971/1962] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:22.332927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.344062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321c282-7a3b-4a97-9868-fed09b7d4b9e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.348873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.350181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.350461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.350730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] PATH_INFO: `/v3/projects/4d14d4209d9d4e7790e362d747e56dd7/users/a89740eb26a947e98eeb358179464e32/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.351500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 992/1963] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:22.352345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.352666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d14d4209d9d4e7790e362d747e56dd7, user_id=a89740eb26a947e98eeb358179464e32, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.354193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.355372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.358804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:22.369896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 972/1964] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:22 2022] PUT /identity/v3/projects/4d14d4209d9d4e7790e362d747e56dd7/users/a89740eb26a947e98eeb358179464e32/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:22.381204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.416545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55687503-c862-4b39-91be-6e02ae9d3bdb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.456351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55ffce4-8f3a-459a-af9a-6d3745422c4f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.461782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.462615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.463037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.463308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] PATH_INFO: `/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904/users/1314e33fd81647cfa5537bd2bcd37e4c/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.465458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba2cbb69e7644d9b0f6d75f86b06904, user_id=1314e33fd81647cfa5537bd2bcd37e4c, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.468756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.470101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.470877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.471226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.471499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.482322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.492510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 973/1965] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:22 2022] PUT /identity/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904/users/1314e33fd81647cfa5537bd2bcd37e4c/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:22.492510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.492739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc0b7c7-c072-4d20-ad21-cc3b978a9482 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.494574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 993/1966] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:22.495861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff6caf-a741-4348-8963-5faba8033609 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.496145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff6caf-a741-4348-8963-5faba8033609 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.496486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff6caf-a741-4348-8963-5faba8033609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.526568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4fff6caf-a741-4348-8963-5faba8033609 None None] MFA Rules not processed for user `a89740eb26a947e98eeb358179464e32`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:22.527808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4fff6caf-a741-4348-8963-5faba8033609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.572581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:22.585953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330dc2ce-d175-4313-b60d-afa202923ca8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.592120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.593037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.593462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.593889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.594649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.596948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.627331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 994/1967] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:22.628682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.628970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.629307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.638833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 974/1968] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/users => generated 298 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:22.639873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.642265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.670082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] MFA Rules not processed for user `1314e33fd81647cfa5537bd2bcd37e4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:22.671529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.721089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224b081-f7f8-4568-81b8-f565768bbe93 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.730406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.732246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.732696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.733157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.749202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.751132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.766357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 975/1969] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:22.768313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.779820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 995/1970] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:22 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:22.800825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e712f47-c71a-4fb1-bf3c-51f5cd8537a4 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.813396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.814074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.814359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.814631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.815277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.826531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.844235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 976/1971] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:22.848584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.848779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.859674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d6374-78b9-42df-bbd8-07e10af4619e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.864197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.865277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.865558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.870468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cd8622-e11c-4c58-b3a4-d6b86f2ed66e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.875870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.877318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.882281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:22.898195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 977/1972] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:22 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:22.909209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.909899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.920895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.923647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] PATH_INFO: `/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4/users/8ee6c8b598914d949c8d103b786f5709/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.953076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:22.955158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:22.979107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78303eff-8aad-4faf-bd53-a29050a9248b service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:22.990031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:22.991340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:22.992382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:22.992842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] PATH_INFO: `/v3/projects/4c22e5faa8494d349aa679fd4e219399` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:22.994580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c22e5faa8494d349aa679fd4e219399)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:22.996957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:22.998593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc67ca8eb2a24a56b441ab163e3ef4f4, user_id=8ee6c8b598914d949c8d103b786f5709, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.004817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.046752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 996/1973] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:22 2022] PUT /identity/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4/users/8ee6c8b598914d949c8d103b786f5709/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:23.050576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.124948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c22e5faa8494d349aa679fd4e219399', 'request_id': 'req-c9ff26e2-4c18-41e5-8318-b3cef98853ed'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.137319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c22e5faa8494d349aa679fd4e219399', 'request_id': 'req-c9ff26e2-4c18-41e5-8318-b3cef98853ed'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.156905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c22e5faa8494d349aa679fd4e219399', 'request_id': 'req-c9ff26e2-4c18-41e5-8318-b3cef98853ed'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.178326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] The token cache is being invalidate because project 4c22e5faa8494d349aa679fd4e219399 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:23.178759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.179697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c9ff26e2-4c18-41e5-8318-b3cef98853ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.181923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 978/1974] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:22 2022] DELETE /identity/v3/projects/4c22e5faa8494d349aa679fd4e219399 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:23.188115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.188400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.189024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.230702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175c1e1-8716-4a4b-aeda-b70a0e178878 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.244879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:23.246156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.384213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:23.439963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.440684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.440965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.441236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.441949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.448497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 979/1975] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:23.448605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.448700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.448700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.459681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 997/1976] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 408 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:23.460400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.462163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f44115-0c26-47d4-89dd-d816bd3520ec None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.467563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.468211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.468489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.468764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] PATH_INFO: `/v3/users/8c16c13f025b4f0898c184904db97a28` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.470135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c16c13f025b4f0898c184904db97a28)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.471977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.475605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.542511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c16c13f025b4f0898c184904db97a28'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.543386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c16c13f025b4f0898c184904db97a28'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.549114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa6f00d-02b5-435d-ba19-9a617b2e277d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.551738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c16c13f025b4f0898c184904db97a28'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.553392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c16c13f025b4f0898c184904db97a28'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.554563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.555188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.555471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.555747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.557535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.563148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-329df697-09e8-48ee-8296-7b03c755ea1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c16c13f025b4f0898c184904db97a28'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:23.576807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:23.589677 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 980/1977] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:23 2022] DELETE /identity/v3/users/8c16c13f025b4f0898c184904db97a28 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:23.591336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26966ae0-188c-4ead-b34d-00b3502e98d5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.591763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26966ae0-188c-4ead-b34d-00b3502e98d5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.592187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26966ae0-188c-4ead-b34d-00b3502e98d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.617563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-26966ae0-188c-4ead-b34d-00b3502e98d5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:23.618975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-26966ae0-188c-4ead-b34d-00b3502e98d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.647921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.651960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.665697 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 998/1978] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:23 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:23.682059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.684030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.698723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897a67a1-eca1-4917-8ebb-df38ce0f00ae service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.706411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 981/1979] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:23.707684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.707974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.708389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.728232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:23.729468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.744178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 982/1980] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:23.744178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.755604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa6f6da-78bf-4f4f-a3bc-e23109a7c419 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.760469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.761214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.761540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.761822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.762732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:23.763119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.764916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.781571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 983/1981] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:23.783368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.786086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:23.789288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.790003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.790313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.790640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] PATH_INFO: `/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4/users/8ee6c8b598914d949c8d103b786f5709/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.794121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc67ca8eb2a24a56b441ab163e3ef4f4, user_id=8ee6c8b598914d949c8d103b786f5709, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.796801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.808718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 999/1982] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:23 2022] PUT /identity/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4/users/8ee6c8b598914d949c8d103b786f5709/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:23.809499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:23.858977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b798ee-384d-48dd-bc2d-8b822cb88bb2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.862037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:23.912763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3138685a-3ed5-4f5a-a471-07f6b96f2705 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:23.917602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.918266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.918678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.919002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.919685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.921817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca07a56-02bf-410c-a5b1-7ccaa762484a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:23.966250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1000/1983] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:23 2022] POST /identity/v3/projects => generated 375 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:23.971015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.971653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.972282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.972769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:23.973491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:23.975005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:23.975145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:23.988692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:23.990811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.005215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 984/1984] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:24.006385 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.020671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4c99e-37ce-4926-b8d4-6ab9740332b0 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.087725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] MFA Rules not processed for user `8ee6c8b598914d949c8d103b786f5709`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:24.089008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.115568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=1,keystone:SELECT=99 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:24.160965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.162502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.162965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.163335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.164529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.167171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.222811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 985/1985] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:24 2022] POST /identity/v3/users => generated 288 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:24.224090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.238776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e2c48-f117-4057-880e-b077ae779fb7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.245295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.246142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.246860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.247233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.248008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.258014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.258424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1001/1986] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:23 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:24.273460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.274961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.281878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 986/1987] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:24.286531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.300326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaad36a-234e-4280-ae9b-59fba5a85bfa admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.308381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.309011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.309289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.309560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.310234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.311887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.342824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 987/1988] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:24.346613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.369427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c8b82f-38cc-4db3-a7e7-1278b076e7c6 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.374727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.376052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.376510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.376988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] PATH_INFO: `/v3/projects/1430d15f6bf941539e0a89581c995820/users/de8c28666c4c468cb73fafea68c820e4/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.390159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d82f09-441f-4a3b-9ec7-1234d3bae238 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.395336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.396888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.397673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.398209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.400517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.401007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1430d15f6bf941539e0a89581c995820, user_id=de8c28666c4c468cb73fafea68c820e4, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.406459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b7a2e4-1be1-426e-be60-b24b127b5b3e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.437234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 988/1989] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:24 2022] PUT /identity/v3/projects/1430d15f6bf941539e0a89581c995820/users/de8c28666c4c468cb73fafea68c820e4/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:24.439473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.439781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.440203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.474611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:24.479352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] MFA Rules not processed for user `de8c28666c4c468cb73fafea68c820e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:24.481642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.501055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.502692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.528568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1002/1990] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:24 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:24.576808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 989/1991] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:24.664310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.667887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.685400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ad1da1-9a19-42b9-9a00-a14cf1cc0183 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.700345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.707534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:24.799658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.800776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.800776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.801491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] PATH_INFO: `/v3/projects/d3f57fcae0634fdfa444e248abde5ee5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.803133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3f57fcae0634fdfa444e248abde5ee5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.805366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.822794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fab27-4007-4254-af33-e153b3c99765 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.826873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:24.827670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:24.828103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:24.828527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:24.838868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:24.840457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:24.852223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 990/1992] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:24.885609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3f57fcae0634fdfa444e248abde5ee5', 'request_id': 'req-0400b553-60ea-44fe-a47e-de6f264fd244'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:24.892560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3f57fcae0634fdfa444e248abde5ee5', 'request_id': 'req-0400b553-60ea-44fe-a47e-de6f264fd244'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:24.902626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3f57fcae0634fdfa444e248abde5ee5', 'request_id': 'req-0400b553-60ea-44fe-a47e-de6f264fd244'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:24.938481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] The token cache is being invalidate because project d3f57fcae0634fdfa444e248abde5ee5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:24.938927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:24.942012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:24.944316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1003/1993] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:24 2022] DELETE /identity/v3/projects/d3f57fcae0634fdfa444e248abde5ee5 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:24.948077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:24.964900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19e8a32-a8dc-49f0-913d-ec911672e1e2 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:24.973017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:25.064750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:25.065622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:25.065907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:25.066259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] PATH_INFO: `/v3/users/1314e33fd81647cfa5537bd2bcd37e4c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:25.067915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1314e33fd81647cfa5537bd2bcd37e4c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:25.071064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:25.129359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1314e33fd81647cfa5537bd2bcd37e4c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.130083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1314e33fd81647cfa5537bd2bcd37e4c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.139732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1314e33fd81647cfa5537bd2bcd37e4c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.140390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1314e33fd81647cfa5537bd2bcd37e4c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.159765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aed91397-781d-4741-8f2f-064f132b7615 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1314e33fd81647cfa5537bd2bcd37e4c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.187864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 991/1994] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:24 2022] DELETE /identity/v3/users/1314e33fd81647cfa5537bd2bcd37e4c => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:25.702784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:25.706171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:25.739772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0400b553-60ea-44fe-a47e-de6f264fd244 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:25.802148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:25.802793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:25.803072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:25.803365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] PATH_INFO: `/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:25.804493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bba2cbb69e7644d9b0f6d75f86b06904)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:25.806878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:25.833118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:25.860884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bba2cbb69e7644d9b0f6d75f86b06904', 'request_id': 'req-ad1bcf78-2319-49ff-89ac-149df7736007'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.869769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bba2cbb69e7644d9b0f6d75f86b06904', 'request_id': 'req-ad1bcf78-2319-49ff-89ac-149df7736007'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.876022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bba2cbb69e7644d9b0f6d75f86b06904', 'request_id': 'req-ad1bcf78-2319-49ff-89ac-149df7736007'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.888997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] The token cache is being invalidate because project bba2cbb69e7644d9b0f6d75f86b06904 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:25.889793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.890107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ad1bcf78-2319-49ff-89ac-149df7736007 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:25.891897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1004/1995] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:25 2022] DELETE /identity/v3/projects/bba2cbb69e7644d9b0f6d75f86b06904 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:25.896649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:25.897155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:25.897242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:25.917010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:25.918250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:25.977984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 992/1996] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:25.985919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f0031-d712-4985-b376-c375d501d606 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:25.986175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f0031-d712-4985-b376-c375d501d606 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:25.986454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f0031-d712-4985-b376-c375d501d606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.007277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-945f0031-d712-4985-b376-c375d501d606 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:26.008489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-945f0031-d712-4985-b376-c375d501d606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:26.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1005/1997] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:26.020838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.031071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49633688-af8f-49bf-8bfd-b50ee0282b15 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.035619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.036219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.036499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.037345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.038008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:26.038326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.039914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.051388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 993/1998] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:02:26.055577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f0031-d712-4985-b376-c375d501d606 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.064741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f0031-d712-4985-b376-c375d501d606 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.069364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.069964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.070272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.070545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.071176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.073065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.101225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1006/1999] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:26.106310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.121228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8eb915-62f1-44c7-8433-234831c5ac01 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.133910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:26.182553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.183173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.183457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.183734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.184409 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.186422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.208024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 994/2000] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:26.212918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.223269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f794e3c-4485-4d6f-98c3-2fdc4481b1f0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.228627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.229273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.229590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.229869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.230591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.232191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.242664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1007/2001] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:26.246629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.257055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e916d3-7806-4bc6-9bcf-cf0a4bf26aa4 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.261719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.262387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.262671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.262951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.263616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.265234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c168b7-73dd-4bc3-8e7a-cdb44bb1b354 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.281667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 995/2002] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:26.286060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.296735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea2bc1f-f9e8-4519-ae7d-e6e91ef4fce2 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.301503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.302215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.302528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.302804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] PATH_INFO: `/v3/projects/779809b2499e43bbaaf1630684e47afe/users/1a8317f7c3f24f93ad54f560dea5f1ae/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.317780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=779809b2499e43bbaaf1630684e47afe, user_id=1a8317f7c3f24f93ad54f560dea5f1ae, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.320495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.335332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1008/2003] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:26 2022] PUT /identity/v3/projects/779809b2499e43bbaaf1630684e47afe/users/1a8317f7c3f24f93ad54f560dea5f1ae/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:26.345660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.346631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.347257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.383003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] MFA Rules not processed for user `1a8317f7c3f24f93ad54f560dea5f1ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:26.384303 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:26.465282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 996/2004] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:26.469705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.557650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1451d3-1426-4cf7-aef7-10ed0ddc77f7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.569708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.570534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.570978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.571316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.572141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.574669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.587615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.589485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:26.606023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1009/2005] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/projects => generated 383 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:26.610004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.646852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:26.651154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca35ac0-65b0-4e50-9920-5996fb67c4f2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.681175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c4276-cbf7-4f68-a647-5df47a3c66cd admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.687785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.688409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.688690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.688958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.689616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.691090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:26.692409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.705922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.706861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.707150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.707429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.710201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:26.722018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1010/2006] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/users => generated 298 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:26.723303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.734027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21820090-3c1d-4174-9df1-83cd578160fd admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.740469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.741228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.741666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.747284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.747284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.747284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.756700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1011/2007] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:26.761146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.777839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf11f5-a62a-44b5-aa49-6bc35cf332eb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.783472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.784894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.785325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.785743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] PATH_INFO: `/v3/projects/2b8ff788dd4b450c893c25fdf25fb903/users/de0a782b8b434c6ab517c29abaed57a5/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.789833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.791370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.801237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 997/2008] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:26.804055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b8ff788dd4b450c893c25fdf25fb903, user_id=de0a782b8b434c6ab517c29abaed57a5, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.806752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.824061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1012/2009] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:26 2022] PUT /identity/v3/projects/2b8ff788dd4b450c893c25fdf25fb903/users/de0a782b8b434c6ab517c29abaed57a5/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:26.828073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.839784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4937f4ac-7f9c-4ef1-8a6a-903800f3afc1 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.911989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.912687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.913038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.913321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.913992 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.915716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d724fd3-5e34-4167-9886-e9662dd86740 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.927759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 998/2010] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:26 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:26.932024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:26.943769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd72c8-8eda-4d85-9ca0-3d3547b64c77 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:26.949572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:26.950383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.950945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.951298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] PATH_INFO: `/v3/projects/2b8ff788dd4b450c893c25fdf25fb903/users/de0a782b8b434c6ab517c29abaed57a5/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:26.953415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b8ff788dd4b450c893c25fdf25fb903, user_id=de0a782b8b434c6ab517c29abaed57a5, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:26.956361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:26.978015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1013/2011] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:26 2022] PUT /identity/v3/projects/2b8ff788dd4b450c893c25fdf25fb903/users/de0a782b8b434c6ab517c29abaed57a5/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:26.982603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:26.983201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:26.983991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:27.026717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] MFA Rules not processed for user `de0a782b8b434c6ab517c29abaed57a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:27.029963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:27.133969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 999/2012] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:26 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:27.145537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:27.147436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:27.209590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fc926d-f8ac-47c0-8b0f-749cd16d2e80 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:27.214034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:27.214759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:27.215050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:27.215358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:27.225115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:27.226575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:27.236690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1014/2013] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:27 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:27.326994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:27.336483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb65f7d-428c-4154-b6ef-13562994018d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:27.340093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:27.340694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:27.340973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:27.341246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:27.353750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:27.398920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:27.400420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f71d82-f20c-45da-b523-a12bf2696339 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:27.409318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1000/2014] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:27.892830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:27.953390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22abde8e-391e-4fe7-8639-7dec3180f70c service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:27.957921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:27.958598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:27.958881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:27.959159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] PATH_INFO: `/v3/users/1a8317f7c3f24f93ad54f560dea5f1ae` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:27.960272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a8317f7c3f24f93ad54f560dea5f1ae)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:27.962219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:27.977846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:28.008420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a8317f7c3f24f93ad54f560dea5f1ae'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.015544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a8317f7c3f24f93ad54f560dea5f1ae'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.016186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a8317f7c3f24f93ad54f560dea5f1ae'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.016872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a8317f7c3f24f93ad54f560dea5f1ae'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.023760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a8317f7c3f24f93ad54f560dea5f1ae'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.041100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1015/2015] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:27 2022] DELETE /identity/v3/users/1a8317f7c3f24f93ad54f560dea5f1ae => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:28.045531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:28.046020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:28.046126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:28.065351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:28.066506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:28.123502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1001/2016] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:28.133063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:28.134501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:28.190788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba76a7b-c0dc-4221-ac12-38ca1b81615a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:28.194555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:28.195168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:28.195566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:28.195851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:28.204867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:28.206365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:28.216058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1016/2017] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:28.524365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:28.526073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:28.544742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f22cc4-ee70-42c2-b3ce-370529716ada None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:28.553310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:28.554122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:28.554408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:28.554777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] PATH_INFO: `/v3/projects/779809b2499e43bbaaf1630684e47afe` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:28.556184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] RBAC: Authorizing `identity:delete_project(project_id=779809b2499e43bbaaf1630684e47afe)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:28.558292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:28.612584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '779809b2499e43bbaaf1630684e47afe', 'request_id': 'req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.618817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '779809b2499e43bbaaf1630684e47afe', 'request_id': 'req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.624963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '779809b2499e43bbaaf1630684e47afe', 'request_id': 'req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.639489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] The token cache is being invalidate because project 779809b2499e43bbaaf1630684e47afe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:28.639784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.640518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.651641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1002/2018] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:28 2022] DELETE /identity/v3/projects/779809b2499e43bbaaf1630684e47afe => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:28.655984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:28.666840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870dc8c-86cb-4a3f-84b3-8f0205955cde service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:28.762918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:28.763880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:28.764229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:28.766111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] PATH_INFO: `/v3/users/de0a782b8b434c6ab517c29abaed57a5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:28.768780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de0a782b8b434c6ab517c29abaed57a5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:28.771433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:28.797069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:28.798800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:28.838791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:28.847079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de0a782b8b434c6ab517c29abaed57a5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.854134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:28.855182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de0a782b8b434c6ab517c29abaed57a5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.855828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de0a782b8b434c6ab517c29abaed57a5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.856529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de0a782b8b434c6ab517c29abaed57a5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.865386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de0a782b8b434c6ab517c29abaed57a5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:28.873704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa1e94b-4697-4885-aa8b-4ff4b3fa8c98 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:28.878102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:28.878728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:28.879015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:28.879294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:28.881061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:28.883254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1017/2019] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:28 2022] DELETE /identity/v3/users/de0a782b8b434c6ab517c29abaed57a5 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:28.947881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:28.949527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d037c696-269b-46e7-8190-89dcb73e05a0 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:28.959757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1003/2020] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:28 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:29.149892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.151474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:29.220637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b87e2e-ebdb-4dba-8900-e482ce5dc710 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.225849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.226496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.226774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.227047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] PATH_INFO: `/v3/projects/2b8ff788dd4b450c893c25fdf25fb903` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.228151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b8ff788dd4b450c893c25fdf25fb903)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.230100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.276885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b8ff788dd4b450c893c25fdf25fb903', 'request_id': 'req-137b3abf-9f05-4767-a538-68710ebc833c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:29.282365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b8ff788dd4b450c893c25fdf25fb903', 'request_id': 'req-137b3abf-9f05-4767-a538-68710ebc833c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:29.289109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b8ff788dd4b450c893c25fdf25fb903', 'request_id': 'req-137b3abf-9f05-4767-a538-68710ebc833c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:29.301142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] The token cache is being invalidate because project 2b8ff788dd4b450c893c25fdf25fb903 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:29.301428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:29.302077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-137b3abf-9f05-4767-a538-68710ebc833c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:29.303815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1018/2021] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:29 2022] DELETE /identity/v3/projects/2b8ff788dd4b450c893c25fdf25fb903 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:29.308779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.309250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.309435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.328308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:29.329485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:29.387934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1004/2022] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:29.396567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.396920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.397256 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.417709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:29.419044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:29.427502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1019/2023] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:29.431801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.441721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef60b7b-d794-4c20-8b9b-b52cf4b06815 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.446924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.447538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.447852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.448085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.448734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:29.449039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.450671 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.462892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1005/2024] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:02:29.467317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.477664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02745d10-e48b-4d68-aa84-e0e5a918df30 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.482820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.483454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.483995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.484269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.485011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.487090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.521527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1020/2025] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:29 2022] POST /identity/v3/projects => generated 369 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:29.525398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.541421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4366e83-c3f2-4bc5-b1c9-757dad59dec4 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.598202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.598811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.599119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.599390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.600024 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.601930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.628861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1006/2026] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:29 2022] POST /identity/v3/users => generated 285 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:29.633347 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.643599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c89c62-2fca-4d69-beb3-6f2087a32382 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.648984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.649880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.650202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.650853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.652425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.662617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1021/2027] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:29.666447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.672855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:29.676854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ecd5bd-caca-450e-ad09-5fd0cab763be admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.682896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.683522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.683802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.684073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.684722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.686358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2ec982-fba1-46e0-a356-3cf50c9444b2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.697672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1007/2028] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:29.702034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.712655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcfa567-5e88-4393-983c-69451834d2dc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:29.717180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:29.717830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.718150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.718428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] PATH_INFO: `/v3/projects/1ed949adee1f461db75dfbc192abd945/users/f74566bec8b54837b55822a45a4543e5/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.720625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:29.735364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ed949adee1f461db75dfbc192abd945, user_id=f74566bec8b54837b55822a45a4543e5, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:29.738043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:29.750976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1022/2029] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:29 2022] PUT /identity/v3/projects/1ed949adee1f461db75dfbc192abd945/users/f74566bec8b54837b55822a45a4543e5/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:29.772966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:29.773263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:29.773540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:29.808786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] MFA Rules not processed for user `f74566bec8b54837b55822a45a4543e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:29.810561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:29.918440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1008/2030] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:29 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:29.930842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:29.932686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:30.011712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e64ffd8-5485-4b40-8a77-86d04fcfd160 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:30.015608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:30.016219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:30.016497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:30.016770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:30.025144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:30.026608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b89c822-5331-40c6-bd55-65d759ec7b74 service glance] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:30.035652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1023/2031] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:29 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:39.915446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=35 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:40.031312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=42,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:56.122304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:56.123713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:56.199878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabdc3fe-f2be-4ea1-904d-49f93e82520d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:56.204943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:56.205786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:56.206599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:56.206599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] PATH_INFO: `/v3/users/eac48c4521b0489a8530766ddc01a0ef` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:56.207670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eac48c4521b0489a8530766ddc01a0ef)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:56.209727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:56.259815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eac48c4521b0489a8530766ddc01a0ef'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.260470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eac48c4521b0489a8530766ddc01a0ef'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.267755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eac48c4521b0489a8530766ddc01a0ef'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.268629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eac48c4521b0489a8530766ddc01a0ef'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.280156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eac48c4521b0489a8530766ddc01a0ef'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.304299 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1009/2032] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:56 2022] DELETE /identity/v3/users/eac48c4521b0489a8530766ddc01a0ef => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:56.316380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:56.316903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:56.317102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:56.337864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:56.339289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:56.403658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1024/2033] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:56.413817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:56.415249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:56.471188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a055b893-43b9-4a38-affe-57a20dda1229 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:56.474891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:56.475496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:56.476304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:56.476576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:56.486052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:56.487592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:56.501483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1010/2034] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:56.766224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:56.768031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:56.780462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1992ee7a-9e16-462f-8365-2a5272bd8d3f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:56.785087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:56.785943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:56.786367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:56.786511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] PATH_INFO: `/v3/projects/3a37a636f0a3487b9978e430ffa96da8` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:56.789683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a37a636f0a3487b9978e430ffa96da8)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:56.796577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:56.847538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a37a636f0a3487b9978e430ffa96da8', 'request_id': 'req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.854069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a37a636f0a3487b9978e430ffa96da8', 'request_id': 'req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.860409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a37a636f0a3487b9978e430ffa96da8', 'request_id': 'req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.873229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] The token cache is being invalidate because project 3a37a636f0a3487b9978e430ffa96da8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:56.873522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.874196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:56.876582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1025/2035] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:56 2022] DELETE /identity/v3/projects/3a37a636f0a3487b9978e430ffa96da8 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:56.879877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:56.890641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4624d-a880-426a-bd18-dd1e8dfeae2a service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:56.910844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:56.951895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:56.952518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:56.952802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:56.953067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] PATH_INFO: `/v3/users/a89740eb26a947e98eeb358179464e32` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:56.954188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] RBAC: Authorizing `identity:delete_user(user_id=a89740eb26a947e98eeb358179464e32)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:56.956090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.002709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a89740eb26a947e98eeb358179464e32'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.003337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a89740eb26a947e98eeb358179464e32'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.009909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a89740eb26a947e98eeb358179464e32'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.010541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a89740eb26a947e98eeb358179464e32'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.017236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a89740eb26a947e98eeb358179464e32'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.034131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1011/2036] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:56 2022] DELETE /identity/v3/users/a89740eb26a947e98eeb358179464e32 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:57.293703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.295600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.315672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a39d794-7bf9-41b2-ae8e-3b671e1bae2b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.384523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.385382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.385801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.386332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] PATH_INFO: `/v3/projects/4d14d4209d9d4e7790e362d747e56dd7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.387735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d14d4209d9d4e7790e362d747e56dd7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.389730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.423507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:57.446968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.447821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d14d4209d9d4e7790e362d747e56dd7', 'request_id': 'req-f10225f4-1cd3-4f93-9e37-712936b1c94b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.448461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.455080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d14d4209d9d4e7790e362d747e56dd7', 'request_id': 'req-f10225f4-1cd3-4f93-9e37-712936b1c94b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.461618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204a994d-d974-4618-bff1-0ab9aa367abf admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.461951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d14d4209d9d4e7790e362d747e56dd7', 'request_id': 'req-f10225f4-1cd3-4f93-9e37-712936b1c94b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.476101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] The token cache is being invalidate because project 4d14d4209d9d4e7790e362d747e56dd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:57.476515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.477238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f10225f4-1cd3-4f93-9e37-712936b1c94b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.478858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1026/2037] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:57 2022] DELETE /identity/v3/projects/4d14d4209d9d4e7790e362d747e56dd7 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:57.483555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a5bd4-34d5-48a0-98ae-1b60c3d67b3d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.484100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a5bd4-34d5-48a0-98ae-1b60c3d67b3d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.484378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598a5bd4-34d5-48a0-98ae-1b60c3d67b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.505582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-598a5bd4-34d5-48a0-98ae-1b60c3d67b3d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:57.507539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-598a5bd4-34d5-48a0-98ae-1b60c3d67b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.521420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.522916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.523270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.523574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] PATH_INFO: `/v3/users/31d292b6801f433e844182e8e4501960` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.524841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] RBAC: Authorizing `identity:delete_user(user_id=31d292b6801f433e844182e8e4501960)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.527018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.576969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1027/2038] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:57.583326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:57.584751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.585122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.585404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.590532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31d292b6801f433e844182e8e4501960'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.592570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31d292b6801f433e844182e8e4501960'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.602020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31d292b6801f433e844182e8e4501960'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.602851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31d292b6801f433e844182e8e4501960'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.618000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:57.618000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.618000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1028/2039] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:02:57.621224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.624336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0094f566-715d-425b-815d-1b21d76b864e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31d292b6801f433e844182e8e4501960'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:57.630299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccca0981-8d13-43c7-bc79-b27f19ecc615 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.635471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.636061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.636336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.636609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.637244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:02:57.637543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.639140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.643342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1012/2040] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:57 2022] DELETE /identity/v3/users/31d292b6801f433e844182e8e4501960 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:57.648094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.648380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.648651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.654011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1029/2041] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:02:57.655443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.666146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a465c5-54f5-4145-84f6-c3a23b20e77f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.668014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:57.669314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.734794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.735418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.735701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.735976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.736607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.738545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.746745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1013/2042] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:57.756435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.757881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:57.767522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1030/2043] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:57 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:57.771564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.794496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:57.816549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d385e-d0f5-48f1-8897-412070fd06f7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.838237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bace1309-37bb-4c4d-9068-5d4b8ce572dd admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.843270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.843884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.844192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.844497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.845128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.847088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.867107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.867717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.867993 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.868262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.868606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1031/2044] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:57 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:57.872591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.877786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.879173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.882958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9cfa51-f32d-427b-a9a8-520f7a8160ff admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.888390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.889015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.889236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1014/2045] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:57.889369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.889554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.890224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.891801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.903120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1032/2046] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:57.915839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.929639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9085c4f-535b-4666-b277-8cc3bc761796 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.934777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.935398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.935797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.936102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.936735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:57.938641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37221be3-3688-4cb2-b441-0bd2589650a8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:57.950741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1015/2047] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:57.954640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:57.965574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b965fab-dc84-40e3-a6f4-ddc9e777af11 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:57.972814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:57.973747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:57.974089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:57.974435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] PATH_INFO: `/v3/projects/9765269cf2ab4cffa8c793e3707cfbf8/users/ccd57e88091e4d04bedf12f67791c9f8/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:57.997019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9765269cf2ab4cffa8c793e3707cfbf8, user_id=ccd57e88091e4d04bedf12f67791c9f8, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.000194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.027819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1033/2048] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:57 2022] PUT /identity/v3/projects/9765269cf2ab4cffa8c793e3707cfbf8/users/ccd57e88091e4d04bedf12f67791c9f8/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:58.039124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.039476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.039828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.063670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:58.079245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] MFA Rules not processed for user `ccd57e88091e4d04bedf12f67791c9f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:58.081416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.169225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1016/2049] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:58.173663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.242583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7d504c-8d6e-46c2-a6e9-c8aed82ecd9d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.248220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.249608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.256065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.256699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.257532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.257803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.258483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.260383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.264541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ce5bb7-8702-4291-b442-5aa6d5f28bd1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.276339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.277404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.277856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.278219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] PATH_INFO: `/v3/projects/09ea94dc884f483dab8cc80ccdf5b935` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.279787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09ea94dc884f483dab8cc80ccdf5b935)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.281668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.289405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1034/2050] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/projects => generated 379 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:58.293350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.331257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09ea94dc884f483dab8cc80ccdf5b935', 'request_id': 'req-1da18fa3-482d-4fca-bdb8-7e8790bce033'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.337362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09ea94dc884f483dab8cc80ccdf5b935', 'request_id': 'req-1da18fa3-482d-4fca-bdb8-7e8790bce033'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.344353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09ea94dc884f483dab8cc80ccdf5b935', 'request_id': 'req-1da18fa3-482d-4fca-bdb8-7e8790bce033'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.356506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b05ec2-00d2-4754-ad4b-a2fda7ef8099 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.356873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] The token cache is being invalidate because project 09ea94dc884f483dab8cc80ccdf5b935 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:58.357269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.357963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.359863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1017/2051] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:58 2022] DELETE /identity/v3/projects/09ea94dc884f483dab8cc80ccdf5b935 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:58.360338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:58.364087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.365762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.416506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.417137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.417320 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.417653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.418221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.420123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.431758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da18fa3-482d-4fca-bdb8-7e8790bce033 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.437543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.438443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.439029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.439262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] PATH_INFO: `/v3/users/eb58a5e0d4654b8797f7e7cb96ed9c46` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.440686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb58a5e0d4654b8797f7e7cb96ed9c46)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.443415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.447508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1035/2052] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/users => generated 290 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:58.448648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.450470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.463292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a140f6a-16c6-4d79-a06a-d5f77a9c04fe admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.468338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.468956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.469239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.469520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.470200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.472094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.475041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:58.487809 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1036/2053] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:58.492985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.503379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd7b1ba-f7c2-4cd4-9d5b-6acbbfe63cbe admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.509018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.509157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb58a5e0d4654b8797f7e7cb96ed9c46'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.509846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.510002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb58a5e0d4654b8797f7e7cb96ed9c46'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.510170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.510457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.511103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.512700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.523459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb58a5e0d4654b8797f7e7cb96ed9c46'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.524114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb58a5e0d4654b8797f7e7cb96ed9c46'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.525695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1037/2054] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:02:58.530152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.531903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-4e6b43a9-d2c3-4a2e-8bc0-2c8b265ef926 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb58a5e0d4654b8797f7e7cb96ed9c46'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:58.532018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.546235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13f7e5-3ecc-4c18-9cba-f6d340a13c21 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.551115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.551352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1018/2055] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:58 2022] DELETE /identity/v3/users/eb58a5e0d4654b8797f7e7cb96ed9c46 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:58.552012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.552300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.552577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] PATH_INFO: `/v3/projects/d123ba0a6df5428681a8ef33ff0f066b/users/842dd0b9ed2644be87bf3f3d51674ede/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.568470 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d123ba0a6df5428681a8ef33ff0f066b, user_id=842dd0b9ed2644be87bf3f3d51674ede, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.571690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.591818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1038/2056] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:58 2022] PUT /identity/v3/projects/d123ba0a6df5428681a8ef33ff0f066b/users/842dd0b9ed2644be87bf3f3d51674ede/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:58.602702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.603067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.603261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.644635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] MFA Rules not processed for user `842dd0b9ed2644be87bf3f3d51674ede`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:58.646918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.777296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1019/2057] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:58.778485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.869455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321f18d-24f7-49b4-afa4-57eccc7a927d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.874666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:58.875492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:58.875836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:58.876178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:58.882675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:58.882675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:58.908846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.910077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:58.923464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1039/2058] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/projects => generated 379 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:02:58.928728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:58.928849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df91467-2885-43bd-9470-6ef33a057bbf None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:58.977593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:59.005500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b1b92-e18b-45ba-b456-cf44adf85152 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.008972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.009689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.010067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.010433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] PATH_INFO: `/v3/projects/cf6327e570ee4dbb83b659ec54da55fb` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.011685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf6327e570ee4dbb83b659ec54da55fb)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.012633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.013247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.013525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.013947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.014232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.014619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.017336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.046916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1040/2059] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:02:58 2022] POST /identity/v3/users => generated 296 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:02:59.048035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.062023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:59.068846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e059205e-b1a8-4ba2-a289-b7725b2f036c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.086508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf6327e570ee4dbb83b659ec54da55fb', 'request_id': 'req-2515e466-bcfc-4974-a9cf-a147ddc17522'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.095804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf6327e570ee4dbb83b659ec54da55fb', 'request_id': 'req-2515e466-bcfc-4974-a9cf-a147ddc17522'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.107594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf6327e570ee4dbb83b659ec54da55fb', 'request_id': 'req-2515e466-bcfc-4974-a9cf-a147ddc17522'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.126143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] The token cache is being invalidate because project cf6327e570ee4dbb83b659ec54da55fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:59.126419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.127072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.129011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1020/2060] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:58 2022] DELETE /identity/v3/projects/cf6327e570ee4dbb83b659ec54da55fb => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:59.133697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.135185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.151966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.153444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.153444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.153444 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.154041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.157421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.166849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1041/2061] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:59.171200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.173076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.205216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9b62e-4b34-41ab-b642-c31ac95994a5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.218168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.218858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.219176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.219451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] PATH_INFO: `/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915/users/2a117fcde84942129d538ab01b913201/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.238528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4e1b3b8dbaf4207bbaf3fb941caa915, user_id=2a117fcde84942129d538ab01b913201, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.242153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.251317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2515e466-bcfc-4974-a9cf-a147ddc17522 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.258642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.259852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.260353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.261587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] PATH_INFO: `/v3/users/8ee6c8b598914d949c8d103b786f5709` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.263370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ee6c8b598914d949c8d103b786f5709)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.263991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1042/2062] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:59 2022] PUT /identity/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915/users/2a117fcde84942129d538ab01b913201/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:59.267484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.268800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.350410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ee6c8b598914d949c8d103b786f5709'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.351821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ee6c8b598914d949c8d103b786f5709'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.364169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8ca643-f611-441d-aae5-bebcb595fa1d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.368078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ee6c8b598914d949c8d103b786f5709'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.368886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ee6c8b598914d949c8d103b786f5709'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.369824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.370596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.370942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.371288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.372225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.373962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.378723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6b131d2f-02fe-4613-9102-68a96ffcdd1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ee6c8b598914d949c8d103b786f5709'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.386172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1043/2063] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:02:59.394381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.396094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.403756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1021/2064] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:02:59 2022] DELETE /identity/v3/users/8ee6c8b598914d949c8d103b786f5709 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:59.409494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff4cce9-3f60-4a9d-a965-9fb9da2fe78b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.442569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:59.472574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.473255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.473535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.473811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] PATH_INFO: `/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915/users/2a117fcde84942129d538ab01b913201/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.476552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4e1b3b8dbaf4207bbaf3fb941caa915, user_id=2a117fcde84942129d538ab01b913201, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.479353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.490123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1044/2065] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:02:59 2022] PUT /identity/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915/users/2a117fcde84942129d538ab01b913201/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:02:59.501932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.502461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.502594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.533381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] MFA Rules not processed for user `2a117fcde84942129d538ab01b913201`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:59.534983 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.571309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:02:59.613057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1022/2066] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:59 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:59.625569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.627332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.689751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f702e4e-8da9-4ef3-8a12-f284b5e83a1f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.691140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.691375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.692101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.692480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.692637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.692944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.707209 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.708640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.721038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1045/2067] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:59 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:02:59.771281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:02:59.781945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a175c98b-46ad-4655-88d8-1ec34ff68a1a service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.786435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.787077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.787353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.787700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.790205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.802480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1597f5-7956-4d58-b451-d8dd5c5a88eb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:02:59.807632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:02:59.808266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.808566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.808837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] PATH_INFO: `/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.810007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc67ca8eb2a24a56b441ab163e3ef4f4)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.811940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.852897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:02:59.854350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158244b-dd5e-45c2-8103-7af85ce0260a service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:02:59.860679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc67ca8eb2a24a56b441ab163e3ef4f4', 'request_id': 'req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.863305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1046/2068] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:02:59 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:02:59.866428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc67ca8eb2a24a56b441ab163e3ef4f4', 'request_id': 'req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.872876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc67ca8eb2a24a56b441ab163e3ef4f4', 'request_id': 'req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.886044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] The token cache is being invalidate because project fc67ca8eb2a24a56b441ab163e3ef4f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:02:59.886361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.887004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-6f2329cc-6ac0-4b6a-8d4a-a6c920b65236 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:02:59.888782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1023/2069] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:02:59 2022] DELETE /identity/v3/projects/fc67ca8eb2a24a56b441ab163e3ef4f4 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:02:59.893739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.894065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.894352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:02:59.916053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:02:59.917351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:02:59.991740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1047/2070] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:02:59.993506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:02:59.993785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:02:59.994083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.017241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:00.018548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:00.028961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1024/2071] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:00.034251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.048205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ddb1d8-afe3-4a36-b1f9-6a9de9535cf7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.059906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.060766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.060880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.061134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.061868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:00.062414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.063839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.067045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:00.081884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1048/2072] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:00.083135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.094098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4129f0ba-ed99-49cd-9507-6f1687689e9c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.106577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.107184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.107468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.107787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.108390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.110622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.152093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1025/2073] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:00 2022] POST /identity/v3/projects => generated 379 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:00.155985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.166994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad396904-c1b2-4ee4-a05e-c71f6e48a602 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.237679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.238364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.238777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.239054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.239760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.241879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.271877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1049/2074] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:00 2022] POST /identity/v3/users => generated 290 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:00.280317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.294359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd9a6bc-a557-42d1-b9f1-e8ac7c5956a9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.299639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.300256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.300540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.300866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.301664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.303509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.325472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.326792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1026/2075] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:00.339172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bb55e-0006-4ef2-8adf-3f66a3d87c3d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.344790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.345417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.345788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.345967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.346648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.348267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6457d741-5c1b-452a-ba2a-58d5d667d7df admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.366338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.370152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1050/2076] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:00.377067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2a8db7-08bf-4bbc-bdb9-a3698e8d7994 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.382440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.383096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.383375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.383647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] PATH_INFO: `/v3/projects/9aff70e0d61c41fbb469ed4bcd785b23/users/6b926f7161c0487a8c2b04ada3b374e3/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.398834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:00.402793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aff70e0d61c41fbb469ed4bcd785b23, user_id=6b926f7161c0487a8c2b04ada3b374e3, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.407111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.422698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1027/2077] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:00 2022] PUT /identity/v3/projects/9aff70e0d61c41fbb469ed4bcd785b23/users/6b926f7161c0487a8c2b04ada3b374e3/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:00.427966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368e95b2-bc63-4cfb-8ea2-b2f104c991cd None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.428322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368e95b2-bc63-4cfb-8ea2-b2f104c991cd None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.428739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368e95b2-bc63-4cfb-8ea2-b2f104c991cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.449239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-368e95b2-bc63-4cfb-8ea2-b2f104c991cd None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:00.450645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-368e95b2-bc63-4cfb-8ea2-b2f104c991cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:00.515587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1051/2078] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:00.530504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:00.532780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:00.607596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6076ca-af01-4388-9c6d-86fba101e183 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:00.611750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:00.612353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:00.612626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:00.612895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:00.622217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:00.623629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:00.633441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1028/2079] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:04.968271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:04.968806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:04.968806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:04.999869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] MFA Rules not processed for user `6b926f7161c0487a8c2b04ada3b374e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:05.001141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:05.070135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1052/2080] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:05.074571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.084891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8152d7d1-ca6a-4360-a83a-6078612340bb service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.094275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.094969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.095317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.095656 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.096364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.098377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.134801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1029/2081] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:05 2022] POST /identity/v3/projects => generated 377 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:05.139047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.146345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:05.154161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0095c1-10d1-4f88-b3d6-6b884e2f52ed None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.213899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.214569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.214859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.215136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.215776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.217724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.241101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1053/2082] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:05 2022] POST /identity/v3/users => generated 295 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:05.245236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.256093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a3e50c-82e8-4242-a06c-f40d2502d477 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.261025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.261648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.261928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.262317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.262969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.264566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.282960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1030/2083] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:05 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:05.287354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.298830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646822c1-fffb-4582-9b37-7c4576ef26b0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.303548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.304215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.304495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.304768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] PATH_INFO: `/v3/projects/0340d1268ba4425396d69ff470e52914/users/b18ba8664afc42e1bdce1f8648f36599/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.319522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0340d1268ba4425396d69ff470e52914, user_id=b18ba8664afc42e1bdce1f8648f36599, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.322226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.336166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1054/2084] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:05 2022] PUT /identity/v3/projects/0340d1268ba4425396d69ff470e52914/users/b18ba8664afc42e1bdce1f8648f36599/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:05.340160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.350260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45f525-876c-4011-9e2b-407703e752a4 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.406676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.407290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.407570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.407846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.408493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.410113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.420248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1031/2085] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:05 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:05.424992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:05.436107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25748a3-834f-4e2b-8244-c4720b56281f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:05.444798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:05.444798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:05.444798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:05.444798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] PATH_INFO: `/v3/projects/0340d1268ba4425396d69ff470e52914/users/b18ba8664afc42e1bdce1f8648f36599/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:05.445164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0340d1268ba4425396d69ff470e52914, user_id=b18ba8664afc42e1bdce1f8648f36599, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:05.447672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39615f81-f8c5-4332-9825-362f96cb3b27 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:05.458211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1055/2086] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:05 2022] PUT /identity/v3/projects/0340d1268ba4425396d69ff470e52914/users/b18ba8664afc42e1bdce1f8648f36599/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:08.449394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:08.452256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:08.480326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b26d85-22d9-47f7-8a60-45e48dec2321 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:08.647528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:08.666674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:08.668053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:08.669686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:08.670287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] PATH_INFO: `/v3/users/ccd57e88091e4d04bedf12f67791c9f8` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:08.672556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccd57e88091e4d04bedf12f67791c9f8)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:08.679528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:08.781040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccd57e88091e4d04bedf12f67791c9f8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:08.782183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccd57e88091e4d04bedf12f67791c9f8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:08.796435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccd57e88091e4d04bedf12f67791c9f8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:08.797543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccd57e88091e4d04bedf12f67791c9f8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:08.820227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccd57e88091e4d04bedf12f67791c9f8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:08.898768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1032/2087] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:08 2022] DELETE /identity/v3/users/ccd57e88091e4d04bedf12f67791c9f8 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:08.903622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f7930-4570-4790-b324-88a3096421eb None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:08.904153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f7930-4570-4790-b324-88a3096421eb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:08.904153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022f7930-4570-4790-b324-88a3096421eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:08.964082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-022f7930-4570-4790-b324-88a3096421eb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:08.966105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-022f7930-4570-4790-b324-88a3096421eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:09.163228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1056/2088] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:09.171212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:09.172827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:09.281898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0ca743-d6b7-4d16-8ea7-321d43717cd5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:09.290023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:09.290838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:09.291147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:09.291490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:09.303874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:09.306076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:09.325003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1033/2089] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:10.002602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022f7930-4570-4790-b324-88a3096421eb None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:10.004275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-022f7930-4570-4790-b324-88a3096421eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:10.040203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022f7930-4570-4790-b324-88a3096421eb None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:10.063850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:10.080122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:10.082275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:10.082803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] PATH_INFO: `/v3/projects/9765269cf2ab4cffa8c793e3707cfbf8` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:10.092761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] RBAC: Authorizing `identity:delete_project(project_id=9765269cf2ab4cffa8c793e3707cfbf8)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:10.114463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:10.261329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9765269cf2ab4cffa8c793e3707cfbf8', 'request_id': 'req-625f57ac-9410-4a52-8115-9ad521dfdbea'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.275710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:10.294515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9765269cf2ab4cffa8c793e3707cfbf8', 'request_id': 'req-625f57ac-9410-4a52-8115-9ad521dfdbea'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.311736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9765269cf2ab4cffa8c793e3707cfbf8', 'request_id': 'req-625f57ac-9410-4a52-8115-9ad521dfdbea'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.363734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] The token cache is being invalidate because project 9765269cf2ab4cffa8c793e3707cfbf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:10.364473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.365689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-625f57ac-9410-4a52-8115-9ad521dfdbea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.368397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1057/2090] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:10 2022] DELETE /identity/v3/projects/9765269cf2ab4cffa8c793e3707cfbf8 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:10.372772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:10.394302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd3b92-1beb-4d1b-973d-57003c979f22 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:10.576136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:10.577435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:10.577435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:10.577663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] PATH_INFO: `/v3/users/842dd0b9ed2644be87bf3f3d51674ede` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:10.578751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=842dd0b9ed2644be87bf3f3d51674ede)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:10.580714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:10.635972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:10.715150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '842dd0b9ed2644be87bf3f3d51674ede'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.718379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '842dd0b9ed2644be87bf3f3d51674ede'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.735732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '842dd0b9ed2644be87bf3f3d51674ede'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.740443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '842dd0b9ed2644be87bf3f3d51674ede'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.755399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '842dd0b9ed2644be87bf3f3d51674ede'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:10.782604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1034/2091] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:10 2022] DELETE /identity/v3/users/842dd0b9ed2644be87bf3f3d51674ede => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:11.259510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:11.260085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:11.262496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:11.309455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] MFA Rules not processed for user `b18ba8664afc42e1bdce1f8648f36599`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:11.315483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:11.362920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:11.364657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:11.426231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1058/2092] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:11 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:11.435407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:11.439682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:11.471217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5eb40-2965-489b-bb51-e6be2669a5b1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:11.476537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:11.477107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:11.477535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:11.477810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] PATH_INFO: `/v3/projects/d123ba0a6df5428681a8ef33ff0f066b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:11.479239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d123ba0a6df5428681a8ef33ff0f066b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:11.481176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:11.531558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57a002-4b25-488e-aec5-c68352440fe1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:11.538491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd123ba0a6df5428681a8ef33ff0f066b', 'request_id': 'req-86b3312c-003a-4132-be2c-728555986ca1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.546614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd123ba0a6df5428681a8ef33ff0f066b', 'request_id': 'req-86b3312c-003a-4132-be2c-728555986ca1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.557561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd123ba0a6df5428681a8ef33ff0f066b', 'request_id': 'req-86b3312c-003a-4132-be2c-728555986ca1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.574009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] The token cache is being invalidate because project d123ba0a6df5428681a8ef33ff0f066b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:11.574306 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.575035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.576572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1035/2093] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:11 2022] DELETE /identity/v3/projects/d123ba0a6df5428681a8ef33ff0f066b => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:11.580448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:11.583554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:11.622704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:11.624149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:11.635024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:11.636178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:11.636990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:11.637505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:11.641017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:11.671105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b3312c-003a-4132-be2c-728555986ca1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:11.676232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:11.676979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:11.677334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:11.677685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] PATH_INFO: `/v3/users/2a117fcde84942129d538ab01b913201` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:11.679825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a117fcde84942129d538ab01b913201)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:11.681852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:11.716701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:11.719067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:11.732734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1059/2094] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:11 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:11.758325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a117fcde84942129d538ab01b913201'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.759019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a117fcde84942129d538ab01b913201'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.768490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a117fcde84942129d538ab01b913201'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.769190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a117fcde84942129d538ab01b913201'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.778874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-39913a0d-ec69-4207-b6de-4714ea752e4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a117fcde84942129d538ab01b913201'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:11.800854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1036/2095] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:11 2022] DELETE /identity/v3/users/2a117fcde84942129d538ab01b913201 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:12.155825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.167767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110885ea-86a3-4a04-abd7-5ed3c047cda6 service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.259102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.260364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.260364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.260364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] PATH_INFO: `/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.261547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4e1b3b8dbaf4207bbaf3fb941caa915)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.263969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.317854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4e1b3b8dbaf4207bbaf3fb941caa915', 'request_id': 'req-605dccd7-e328-4ec5-831a-de6a0b9d666c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:12.329332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4e1b3b8dbaf4207bbaf3fb941caa915', 'request_id': 'req-605dccd7-e328-4ec5-831a-de6a0b9d666c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:12.339420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4e1b3b8dbaf4207bbaf3fb941caa915', 'request_id': 'req-605dccd7-e328-4ec5-831a-de6a0b9d666c'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:12.361065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] The token cache is being invalidate because project d4e1b3b8dbaf4207bbaf3fb941caa915 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:12.361375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:12.366673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-605dccd7-e328-4ec5-831a-de6a0b9d666c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:12.373070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1060/2096] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:12 2022] DELETE /identity/v3/projects/d4e1b3b8dbaf4207bbaf3fb941caa915 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:12.385004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.385473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.385755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.408198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:12.409416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:12.491801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1037/2097] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:12.499992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.500339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.500623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.532510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:12.534132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:12.544887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1061/2098] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:12.549768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.563548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a0391f-8a2d-4515-9b2d-1566d2a6302a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.569991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.570920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.571292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.571680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.572544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:12.572950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.577017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.598917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1038/2099] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:12.602222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.616655 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393ee1b2-f7c6-485a-affc-45160aaef559 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.623928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.625216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.626340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.626894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.627915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.630374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.672352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1062/2100] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:12 2022] POST /identity/v3/projects => generated 385 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:12.677117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.688735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc54d9-3d93-4fd5-a758-f156323eb055 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.725206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:12.759102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.759799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.760079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.760396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.761176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.763440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.790381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1039/2101] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:12 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:12.794680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.802425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:12.809640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8738f033-9dd0-43b0-ac24-99310d9512b9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.817858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.818533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.820683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.820900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.821760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.824275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.836501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1063/2102] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:12.840241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.851446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe1e11-3517-4542-a101-34c2e18653c2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.856188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.857293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.857293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.857739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.858394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.860294 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df6b263-210c-4a29-ab8a-42ad0c4713d6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.875916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1040/2103] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:12.877100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:12.887882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34231c51-76ed-4325-8b70-b93eef8007f6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:12.892822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:12.893499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.893781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.894086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] PATH_INFO: `/v3/projects/3c3084acb50e4cefac4534ce51276a86/users/ed0f48ea87044ba9b89278061a1ab369/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.915354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c3084acb50e4cefac4534ce51276a86, user_id=ed0f48ea87044ba9b89278061a1ab369, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:12.918638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:12.937288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1064/2104] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:12 2022] PUT /identity/v3/projects/3c3084acb50e4cefac4534ce51276a86/users/ed0f48ea87044ba9b89278061a1ab369/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:12.948721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:12.949002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:12.949274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:12.994154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] MFA Rules not processed for user `ed0f48ea87044ba9b89278061a1ab369`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:12.995713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:13.103499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1041/2105] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:12 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:13.107426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.176315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9866152-4ec8-4b28-8a5a-ffc19f97bc97 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.181571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.182215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.182496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.182771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.183420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.185317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.215616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1065/2106] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:13 2022] POST /identity/v3/projects => generated 387 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:13.219684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.231490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb6fb9b-32f0-4182-af5b-e08d1e351fc8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.306417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.307277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.307561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.307832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.308525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.310406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.335293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1042/2107] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:13 2022] POST /identity/v3/users => generated 300 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:13.339391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.349964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3fad06-1967-46b2-aa99-f749915e9348 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.355864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.357375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.357860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.358334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.359618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.361386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.372909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1066/2108] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:13.377515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.391777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c7629-8d3f-4599-9c8d-5ef395ab3a49 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.410549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.415276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.415575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.415860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] PATH_INFO: `/v3/projects/b64b7660ae7e4f919a42aeba59878321/users/1df6144a43704a689340118bdfdc0470/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.434642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b64b7660ae7e4f919a42aeba59878321, user_id=1df6144a43704a689340118bdfdc0470, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.437569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.464504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1043/2109] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:13 2022] PUT /identity/v3/projects/b64b7660ae7e4f919a42aeba59878321/users/1df6144a43704a689340118bdfdc0470/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:13.469468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.482731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745e674-8e80-4889-babc-7057f682b766 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.556682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.557321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.557670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.558030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.558785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.560416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978e49e-051d-48b5-812e-a58fb9c28d32 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.577138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1067/2110] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:13.578719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.606805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285e76dc-40ee-4ca3-8558-3a21a5462f89 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.619653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.620404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.620751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.621091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] PATH_INFO: `/v3/projects/b64b7660ae7e4f919a42aeba59878321/users/1df6144a43704a689340118bdfdc0470/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.623769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b64b7660ae7e4f919a42aeba59878321, user_id=1df6144a43704a689340118bdfdc0470, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.627582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.631126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:13.648829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1044/2111] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:13 2022] PUT /identity/v3/projects/b64b7660ae7e4f919a42aeba59878321/users/1df6144a43704a689340118bdfdc0470/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:13.660538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.660822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.661096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.690680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] MFA Rules not processed for user `1df6144a43704a689340118bdfdc0470`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:13.691893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:13.720713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:13.782073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1068/2112] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:13 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:13.793769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:13.796378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:13.868528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0419d745-905a-4ec0-b15c-3fd3ffcb68d5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:13.873075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:13.874296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:13.874768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:13.875117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:13.890009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:13.891505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fc8ea7-2f93-4c40-a353-8ead3a9f6f1c service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:13.903194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1045/2113] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:13 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:14.599958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:14.671586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2722d7-3b76-462a-a67f-0c14164a7556 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:14.677713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:14.679004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:14.679332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:14.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] PATH_INFO: `/v3/users/ed0f48ea87044ba9b89278061a1ab369` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:14.680748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed0f48ea87044ba9b89278061a1ab369)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:14.682711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:14.735977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed0f48ea87044ba9b89278061a1ab369'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:14.745012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed0f48ea87044ba9b89278061a1ab369'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:14.745812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed0f48ea87044ba9b89278061a1ab369'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:14.746445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed0f48ea87044ba9b89278061a1ab369'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:14.754860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed0f48ea87044ba9b89278061a1ab369'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:14.788890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1069/2114] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:14 2022] DELETE /identity/v3/users/ed0f48ea87044ba9b89278061a1ab369 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:14.796413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:14.796928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:14.796928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:14.824237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:14.825454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:14.898063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1046/2115] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:14.909665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:14.914317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:15.013897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:15.032402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1742da46-feed-49b8-babb-740dbe05fc25 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:15.038178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:15.039623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:15.039623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:15.039834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:15.052432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:15.054709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:15.078751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1070/2116] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:15.493323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:15.502068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:15.535610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834be5b-7e59-497a-abf6-e7a2bfdcf950 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:15.544687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:15.545334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:15.545611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:15.545882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] PATH_INFO: `/v3/projects/3c3084acb50e4cefac4534ce51276a86` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:15.547699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c3084acb50e4cefac4534ce51276a86)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:15.550514 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:15.628698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c3084acb50e4cefac4534ce51276a86', 'request_id': 'req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.635816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c3084acb50e4cefac4534ce51276a86', 'request_id': 'req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.643025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c3084acb50e4cefac4534ce51276a86', 'request_id': 'req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.666099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] The token cache is being invalidate because project 3c3084acb50e4cefac4534ce51276a86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:15.666404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.668948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.671443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1047/2117] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:15 2022] DELETE /identity/v3/projects/3c3084acb50e4cefac4534ce51276a86 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:15.676983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:15.692195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e03d2d-cce9-474f-9248-dc2789f62440 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:15.854754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:15.855443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:15.855744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:15.855977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] PATH_INFO: `/v3/users/1df6144a43704a689340118bdfdc0470` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:15.863638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1df6144a43704a689340118bdfdc0470)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:15.866858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:15.927381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1df6144a43704a689340118bdfdc0470'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.940044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1df6144a43704a689340118bdfdc0470'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.941184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1df6144a43704a689340118bdfdc0470'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.942018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1df6144a43704a689340118bdfdc0470'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.956259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca064c20-198b-4009-b9ce-c721ae3a4e79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1df6144a43704a689340118bdfdc0470'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:15.979242 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1071/2118] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:15 2022] DELETE /identity/v3/users/1df6144a43704a689340118bdfdc0470 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:16.418063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.419503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:16.433883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be5cb4f-de3d-4b92-8dcb-b6d5ce97d354 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.453063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:16.498115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.498756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.499040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.499330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] PATH_INFO: `/v3/projects/b64b7660ae7e4f919a42aeba59878321` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.500433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b64b7660ae7e4f919a42aeba59878321)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.502400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:16.550965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b64b7660ae7e4f919a42aeba59878321', 'request_id': 'req-f7ac6eef-f861-4e56-ad35-543aa4de58c8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:16.556785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b64b7660ae7e4f919a42aeba59878321', 'request_id': 'req-f7ac6eef-f861-4e56-ad35-543aa4de58c8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:16.562965 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b64b7660ae7e4f919a42aeba59878321', 'request_id': 'req-f7ac6eef-f861-4e56-ad35-543aa4de58c8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:16.575040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] The token cache is being invalidate because project b64b7660ae7e4f919a42aeba59878321 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:16.575336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:16.575959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f7ac6eef-f861-4e56-ad35-543aa4de58c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:16.580817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1048/2119] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:16 2022] DELETE /identity/v3/projects/b64b7660ae7e4f919a42aeba59878321 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:16.582425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.582713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.583016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.604267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:16.605471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:16.667153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1072/2120] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:16.675278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.675681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.676057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.697252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:16.698530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:16.706703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1049/2121] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:16.711701 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.717901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:16.721729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b3af7-6166-403d-87cb-1bae69f24a5d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.727482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.728062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.728853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.729167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.729905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:16.730363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.732539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:16.745793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1073/2122] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:16.749142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.759442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95263ced-7c25-4311-a379-2ce3f3588f22 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.764115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.764708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.764990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.765272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.765915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.767861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:16.797298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1050/2123] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:16 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:16.800804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.812188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11425df6-3a0a-4b1c-b6f8-db5258e64839 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.880907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.881693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.882110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.882512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.883253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.885412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:16.930782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1074/2124] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:16 2022] POST /identity/v3/users => generated 286 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:16.934651 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.946529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3bb0d-dae8-47fe-aee8-6f9cdc4848c5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.951894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.952688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.952967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.953245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.953899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.955666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:16.969829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1051/2125] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:16.973627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:16.986428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6887db1e-a384-40b0-a563-b0e42b538a25 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:16.992084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:16.992755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:16.993035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:16.993489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:16.994339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:16.995945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881e4f1-99d1-4423-bac1-f44a1ff4c4e9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:17.010965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1075/2126] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:17.016649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:17.027327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1e3d-f329-478f-9271-7f7bde538003 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:17.032401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:17.033034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:17.033316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:17.033591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] PATH_INFO: `/v3/projects/6812aa0ec7234889b3702d8fab9481e6/users/be29d5d3ad50403da073b824eb91adab/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:17.053121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=6812aa0ec7234889b3702d8fab9481e6, user_id=be29d5d3ad50403da073b824eb91adab, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:17.056687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:17.072193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1052/2127] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:17 2022] PUT /identity/v3/projects/6812aa0ec7234889b3702d8fab9481e6/users/be29d5d3ad50403da073b824eb91adab/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:17.078776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c55d40c-ed7e-462e-8b4d-dde8115adf8c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:17.079098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c55d40c-ed7e-462e-8b4d-dde8115adf8c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:17.079376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c55d40c-ed7e-462e-8b4d-dde8115adf8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:17.099034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-1c55d40c-ed7e-462e-8b4d-dde8115adf8c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:17.100839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1c55d40c-ed7e-462e-8b4d-dde8115adf8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:17.165211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1076/2128] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:17.175052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:17.177079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:17.240341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:17.252379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08afad7-e01a-4968-b261-602d3b71afaa admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:17.256509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:17.257307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:17.257594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:17.257873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:17.268278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:17.269766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:17.280213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1053/2129] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:17.945926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:17.946437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:17.946606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:17.977894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] MFA Rules not processed for user `be29d5d3ad50403da073b824eb91adab`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:17.979451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:18.059467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:18.069517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1077/2130] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:17 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:18.070891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.089911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48d2392-04fe-4341-8837-14d069250082 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.095503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.096214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.096491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.096822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.097588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.099578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.131678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1054/2131] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:18 2022] POST /identity/v3/projects => generated 369 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:18.133268 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.144693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3654ba-5790-4aa0-9317-03330e0a8998 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.204791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.206036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.206326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.206602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.207244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.209158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.234870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1078/2132] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:18 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:18.238946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.261086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8331fd-d70c-434c-8f0d-fabde42b12a8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.265946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.267194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.267477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.267754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.268531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.270265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.286770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1055/2133] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:18.292631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.302852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aab57c5-d711-4256-836a-180100a9baa8 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.308056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.308692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.308969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.309241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] PATH_INFO: `/v3/projects/7ffa10cfc4ce4d94ab860636aa026789/users/6f68bda3a203429aae4cf156c7adcadc/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.327219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ffa10cfc4ce4d94ab860636aa026789, user_id=6f68bda3a203429aae4cf156c7adcadc, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.329960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.342503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1079/2134] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:18 2022] PUT /identity/v3/projects/7ffa10cfc4ce4d94ab860636aa026789/users/6f68bda3a203429aae4cf156c7adcadc/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:18.346419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.356829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65509757-387b-496c-a2ef-52f65a4c4f77 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.412865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.413480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.413758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.414064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.414704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.416287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfab5255-8451-42b8-a98e-6f0120c1eeed admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.425670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1056/2135] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:18.430476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:18.440487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ea0bb-2421-4fd6-9ec5-0c9dc12f815f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:18.445119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:18.445790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:18.446106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:18.446380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] PATH_INFO: `/v3/projects/7ffa10cfc4ce4d94ab860636aa026789/users/6f68bda3a203429aae4cf156c7adcadc/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:18.448268 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ffa10cfc4ce4d94ab860636aa026789, user_id=6f68bda3a203429aae4cf156c7adcadc, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:18.451050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:18.461449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1080/2136] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:18 2022] PUT /identity/v3/projects/7ffa10cfc4ce4d94ab860636aa026789/users/6f68bda3a203429aae4cf156c7adcadc/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:19.250629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:19.251155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:19.251254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:19.283880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] MFA Rules not processed for user `6f68bda3a203429aae4cf156c7adcadc`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:19.285109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:19.362871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1057/2137] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:19 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:19.378239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:19.380094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:19.444531 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d500a6c6-acc7-4be6-b1f4-3be2ed86a2a7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:19.448215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:19.448833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:19.449105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:19.449375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:19.507726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:19.509523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:19.519723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1081/2138] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:19 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:25.266552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:25.284186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea838ce4-690a-418f-91f9-e4fe985dbba8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:25.288801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:25.289233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:25.289704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:25.290003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:25.300041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:25.301485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf446766-f344-41f5-9ac5-7030243a1f94 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:25.311253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1058/2139] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:25 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:25.696830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:25.701917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:25.712261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:25.770879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287fc0-b897-41f5-9759-92d1c891bda6 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:25.776364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:25.777020 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:25.777318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:25.777580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] PATH_INFO: `/v3/users/de8c28666c4c468cb73fafea68c820e4` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:25.778752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de8c28666c4c468cb73fafea68c820e4)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:25.780717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:25.830428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de8c28666c4c468cb73fafea68c820e4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:25.839116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de8c28666c4c468cb73fafea68c820e4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:25.839910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de8c28666c4c468cb73fafea68c820e4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:25.840594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de8c28666c4c468cb73fafea68c820e4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:25.848195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de8c28666c4c468cb73fafea68c820e4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:25.866757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1082/2140] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:25 2022] DELETE /identity/v3/users/de8c28666c4c468cb73fafea68c820e4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:25.871667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:25.871959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:25.872235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:25.895084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:25.896243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:25.919766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:25.963330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1059/2141] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:25.976168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:25.977778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.043302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7726409c-cf39-49cd-9cc0-08310e4ec8d9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.046968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.047584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.047863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.048129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.058702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.060497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cbe2b2-ff55-43de-bf68-04c52ebcdda6 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.072382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1083/2142] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:26.216665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.218350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.231067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d58ac72-83ab-475a-b28d-6c553f730bbe None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.237226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.238015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.238366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.238707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] PATH_INFO: `/v3/users/cf13b506718c4b7bbd3fe3a2baf03ed2` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.239944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf13b506718c4b7bbd3fe3a2baf03ed2)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.242561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.312888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf13b506718c4b7bbd3fe3a2baf03ed2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.313585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf13b506718c4b7bbd3fe3a2baf03ed2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.323730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf13b506718c4b7bbd3fe3a2baf03ed2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.324448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf13b506718c4b7bbd3fe3a2baf03ed2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.334414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf13b506718c4b7bbd3fe3a2baf03ed2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.355036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1060/2143] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:26 2022] DELETE /identity/v3/users/cf13b506718c4b7bbd3fe3a2baf03ed2 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:26.364160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.364160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.364160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.384383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:26.385602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.412235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:26.466041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1084/2144] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:26.476650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.477887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.559042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5aaef6-1fb0-4f42-9911-bc577b9feef1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.562839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.563458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.563763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.564049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.574275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.576029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fd1534-af2e-40f6-8380-be62b6e0f877 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.585100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.586335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.598152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1061/2145] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:26.598813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea3197-c20f-41ed-9749-95573616642b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.603970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.604616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.605098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.605371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] PATH_INFO: `/v3/projects/1430d15f6bf941539e0a89581c995820` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.606517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1430d15f6bf941539e0a89581c995820)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.608418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.663107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1430d15f6bf941539e0a89581c995820', 'request_id': 'req-94b2f26e-9422-4b0c-a191-5894e98d0f70'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.679358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1430d15f6bf941539e0a89581c995820', 'request_id': 'req-94b2f26e-9422-4b0c-a191-5894e98d0f70'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.686485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1430d15f6bf941539e0a89581c995820', 'request_id': 'req-94b2f26e-9422-4b0c-a191-5894e98d0f70'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.701576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] The token cache is being invalidate because project 1430d15f6bf941539e0a89581c995820 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:26.702271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.703272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-94b2f26e-9422-4b0c-a191-5894e98d0f70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.704820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1085/2146] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:26 2022] DELETE /identity/v3/projects/1430d15f6bf941539e0a89581c995820 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:26.711237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.711599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.712015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.732916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:26.734572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.775925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:26.803834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1062/2147] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:26.809798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.810408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.810408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.831353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:26.833141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.842663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1086/2148] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:26.846675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.856812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20128fd-7fb3-4d5a-8823-97b3b1b3ffaa None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.861669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.862371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.862715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.863051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.863756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:26.864125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.865805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.878530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1063/2149] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:26.882779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.892640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba7d78b-64a4-4296-9939-a709c1cf870b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.898617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.899337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.899725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.900062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.900837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.902898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.904149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.906001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:26.923154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769facbb-ee6f-445a-ba6d-930b3379fc87 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:26.928449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:26.929106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:26.929507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:26.929798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] PATH_INFO: `/v3/projects/74a16acf93654b72be2765fac8e3141f` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:26.930962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74a16acf93654b72be2765fac8e3141f)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:26.932887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:26.939899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1087/2150] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:26 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:26.940888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:26.986206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74a16acf93654b72be2765fac8e3141f', 'request_id': 'req-53f0d722-8fec-4e61-88be-b31d128169b2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:26.991905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74a16acf93654b72be2765fac8e3141f', 'request_id': 'req-53f0d722-8fec-4e61-88be-b31d128169b2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.001777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74a16acf93654b72be2765fac8e3141f', 'request_id': 'req-53f0d722-8fec-4e61-88be-b31d128169b2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.012723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc92292-5ed8-40f6-9e89-4ad7cff57c41 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.014346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] The token cache is being invalidate because project 74a16acf93654b72be2765fac8e3141f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:27.014644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.015284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.016776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1064/2151] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:26 2022] DELETE /identity/v3/projects/74a16acf93654b72be2765fac8e3141f => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:27.020848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.023484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.043441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:27.076556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.077179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.077487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.077765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.078423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.081064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.096952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f0d722-8fec-4e61-88be-b31d128169b2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.101612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.102283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.102565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.102836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] PATH_INFO: `/v3/users/0da3efae62bb421287615c2a1207935c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.103243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1088/2152] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:26 2022] POST /identity/v3/users => generated 287 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:27.107136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0da3efae62bb421287615c2a1207935c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.107372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.109475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.109877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.123956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998ba660-44aa-4afd-9198-487fef912b05 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.129259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.129876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.130197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.130478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.132317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.133913 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.146033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1089/2153] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:27.151903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.162840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa5e655-a4ed-41bb-9ca8-8b877814ecf9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.167125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0da3efae62bb421287615c2a1207935c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.167600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.167741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0da3efae62bb421287615c2a1207935c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.168203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.168480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.168863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.169507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.171291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.175652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0da3efae62bb421287615c2a1207935c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.176566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0da3efae62bb421287615c2a1207935c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.177955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:27.183928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1090/2154] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:27.184984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd2de38a-d83d-4575-bc08-1c8bfdda7206 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0da3efae62bb421287615c2a1207935c'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.188158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.189955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.208508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0bd401-46bd-4a5e-8e40-673aaaf5cdab admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.208795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1065/2155] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:27 2022] DELETE /identity/v3/users/0da3efae62bb421287615c2a1207935c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:27.283171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.283865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.284148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.284429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] PATH_INFO: `/v3/projects/41719b31f9d84e4e9f8bc17dba294a6b/users/de364365f46c4cf799f875ae9aa9c9d2/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.304854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=41719b31f9d84e4e9f8bc17dba294a6b, user_id=de364365f46c4cf799f875ae9aa9c9d2, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.307585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.322637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1091/2156] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:27 2022] PUT /identity/v3/projects/41719b31f9d84e4e9f8bc17dba294a6b/users/de364365f46c4cf799f875ae9aa9c9d2/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:27.333242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.333728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.333817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.370129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] MFA Rules not processed for user `de364365f46c4cf799f875ae9aa9c9d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:27.371361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.455195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1066/2157] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:27.468174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.470884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.546092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.546800 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.561046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d3f89-3b1c-4089-8702-df0c37c5e4fc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.567321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.567957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.568246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.568527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.607495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.609414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.622833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1092/2158] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:27.635290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6424e1ab-5878-4f42-bbeb-99d61b84f635 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.649264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.650766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.651926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.651926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.652097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] PATH_INFO: `/v3/projects/8808507d6ba949ccb3f08bd7f609d197` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.653112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8808507d6ba949ccb3f08bd7f609d197)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.655151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.661707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9dad3a-1670-46bd-8ef9-36ef0aa00a5f service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.667811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.668448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.668748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.669023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] PATH_INFO: `/v3/users/de364365f46c4cf799f875ae9aa9c9d2` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.671205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de364365f46c4cf799f875ae9aa9c9d2)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.673826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.700244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:INSERT=4,keystone:DELETE=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:27.712362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8808507d6ba949ccb3f08bd7f609d197', 'request_id': 'req-22f5e3b3-0757-4314-b8e5-14236c232fb1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.722361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8808507d6ba949ccb3f08bd7f609d197', 'request_id': 'req-22f5e3b3-0757-4314-b8e5-14236c232fb1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.733328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8808507d6ba949ccb3f08bd7f609d197', 'request_id': 'req-22f5e3b3-0757-4314-b8e5-14236c232fb1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.735909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:27.764856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] The token cache is being invalidate because project 8808507d6ba949ccb3f08bd7f609d197 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:27.765220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.767046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-22f5e3b3-0757-4314-b8e5-14236c232fb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.768335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de364365f46c4cf799f875ae9aa9c9d2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.772157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1067/2159] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:27 2022] DELETE /identity/v3/projects/8808507d6ba949ccb3f08bd7f609d197 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:27.774295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a721d0-a15f-4426-b204-3a5f3f16ba83 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.774929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a721d0-a15f-4426-b204-3a5f3f16ba83 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.775887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a721d0-a15f-4426-b204-3a5f3f16ba83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.776692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de364365f46c4cf799f875ae9aa9c9d2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.778134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de364365f46c4cf799f875ae9aa9c9d2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.779527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de364365f46c4cf799f875ae9aa9c9d2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.791804 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-70ebd8fa-e59a-4f52-9396-5d36de0de111 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de364365f46c4cf799f875ae9aa9c9d2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:27.797933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-12a721d0-a15f-4426-b204-3a5f3f16ba83 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:27.799290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-12a721d0-a15f-4426-b204-3a5f3f16ba83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.814173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1093/2160] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:27 2022] DELETE /identity/v3/users/de364365f46c4cf799f875ae9aa9c9d2 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:27.818498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.819402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.819617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.839558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:27.840907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.867610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1068/2161] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:27.875575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.875860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.876132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.896630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:27.898937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.911869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1094/2162] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:27.921569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.923581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:27.969046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1069/2163] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:27.972914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:27.981505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde95d93-33c3-489e-bb3f-bf6d97434ded None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.986672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.989253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.989253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.989480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1cd8f5-b47e-4adb-9870-e80a1e7e78f4 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:27.989584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:27.989584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:27.989584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:27.990365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:27.994982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:27.995776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:27.996003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:27.996468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.003218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1070/2164] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:28.008123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.008949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.010453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.017876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273c6329-3503-48b2-a5cb-87a6651bae0b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.020175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1095/2165] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:28.023032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.023629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.023903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.024169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.024794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.026985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.055522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1071/2166] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:28.059623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.071045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9831bd6e-4fe7-4071-bbdf-a39e9b81bed1 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.125766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:28.151327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.151907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.152272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.152590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.153353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.155499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.182515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1096/2167] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/users => generated 285 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:28.186846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.199439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f61d27-6790-4a3b-ac8b-1e90d162a6e3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.215692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.217395 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.217828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.218170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.219133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.222930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.243318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1072/2168] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:28.253607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.265518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910e79c-31b9-416f-b152-03d8932c4ebb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.271461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.274261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.274427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.275227 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.275415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.276990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.290169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1097/2169] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:28.295473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.309505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1241f51-edcc-49e0-8853-a0273f8dd5a0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.314691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.315612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.315822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.316115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] PATH_INFO: `/v3/projects/7c6b8983a0cb4ccebf0ad1c5ab51e4f9/users/d6df4ccfa3684404aa833b374f38b1d4/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.335811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c6b8983a0cb4ccebf0ad1c5ab51e4f9, user_id=d6df4ccfa3684404aa833b374f38b1d4, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.340978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d7c74b-513b-4788-a66b-f25a8a98e60d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.359402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1073/2170] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:28 2022] PUT /identity/v3/projects/7c6b8983a0cb4ccebf0ad1c5ab51e4f9/users/d6df4ccfa3684404aa833b374f38b1d4/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:28.375246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.377568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.378219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.378219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.379924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:28.405376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c99fc84-d7cb-49b7-8133-4a753f43019a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.440460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] MFA Rules not processed for user `d6df4ccfa3684404aa833b374f38b1d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:28.442242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:28.459253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:28.495489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.496191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.496474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.496750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] PATH_INFO: `/v3/projects/41719b31f9d84e4e9f8bc17dba294a6b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.498324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41719b31f9d84e4e9f8bc17dba294a6b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.500676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.559059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1074/2171] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:28.563873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.576173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41719b31f9d84e4e9f8bc17dba294a6b', 'request_id': 'req-8835166b-3a97-404c-a963-e3a287483356'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:28.579687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698545f6-c3c3-44d9-bf84-3f2bd7a0dfc6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.583075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41719b31f9d84e4e9f8bc17dba294a6b', 'request_id': 'req-8835166b-3a97-404c-a963-e3a287483356'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:28.591099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41719b31f9d84e4e9f8bc17dba294a6b', 'request_id': 'req-8835166b-3a97-404c-a963-e3a287483356'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:28.653366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] The token cache is being invalidate because project 41719b31f9d84e4e9f8bc17dba294a6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:28.653964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:28.658192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8835166b-3a97-404c-a963-e3a287483356 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:28.660872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1098/2172] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:28 2022] DELETE /identity/v3/projects/41719b31f9d84e4e9f8bc17dba294a6b => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:28.666713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befb907-e026-40a8-b30b-0729f63399c5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.666991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befb907-e026-40a8-b30b-0729f63399c5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.667377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befb907-e026-40a8-b30b-0729f63399c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.699627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.700267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.700710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.701033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.701690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.708220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.711464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2befb907-e026-40a8-b30b-0729f63399c5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:28.715045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2befb907-e026-40a8-b30b-0729f63399c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:28.750034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1099/2173] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:28.758201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.759070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.759660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.811499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1075/2174] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/projects => generated 369 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:28.815364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.818108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:28.826395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:28.827585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:28.889997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:28.923712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1100/2175] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:28.928041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:28.961629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d644e2-e5c2-44f9-9517-04a953977fee None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.967431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d4d716-df07-4490-a4a2-376ad4e56801 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:28.972855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.973483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.973780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.974096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.974735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.976136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:28.977254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:28.978935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:28.981116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:28.981827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:28.982990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:28.984716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:28.987007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.015443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1101/2176] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:29.019071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.019726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1076/2177] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:28 2022] POST /identity/v3/users => generated 291 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:29.026891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.043620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb783480-ad35-46af-8935-594b61283046 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.046304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ba144-a7ae-42ac-ab79-54210f4ba965 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.054611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.058235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.058437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.058714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.060131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.062438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.062662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.064421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.064739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.066324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.071187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.077446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.081068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1077/2178] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:29.082280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.099750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f126f083-b704-4c7c-bffa-452b96964ef8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.114782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.115474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.118343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.118906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] PATH_INFO: `/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992/users/e4226354d2f94230aaa26ec682b632de/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.137850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:29.145591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=10f6d3c0a0ca4c18989e9c2e04e6a992, user_id=e4226354d2f94230aaa26ec682b632de, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.148815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.161017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1102/2179] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:29 2022] POST /identity/v3/projects => generated 373 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:29.168222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.188925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1078/2180] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:29 2022] PUT /identity/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992/users/e4226354d2f94230aaa26ec682b632de/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:29.194941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.261823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276233a4-1daa-4a4a-b966-a56c0e7cc4f9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.351308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c676f2-8d7f-46ea-b674-bb3951db1a5b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.357359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.358627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.358627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.359265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.360114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.362477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.375692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1079/2181] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:29.379820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.385133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.385767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.386083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.386362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.387056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.389005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a1ea37-590a-4370-8054-d412c4cff5b9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.395837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b58c0c7-a8ec-4aac-b90c-062c046ae2d1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.403289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.404400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.404731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.405222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] PATH_INFO: `/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992/users/e4226354d2f94230aaa26ec682b632de/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.408389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=10f6d3c0a0ca4c18989e9c2e04e6a992, user_id=e4226354d2f94230aaa26ec682b632de, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.411849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.424870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1080/2182] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:29 2022] PUT /identity/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992/users/e4226354d2f94230aaa26ec682b632de/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:29.434731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1103/2183] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:29 2022] POST /identity/v3/users => generated 287 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:29.439354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.443477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.447979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.450083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.504613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] MFA Rules not processed for user `e4226354d2f94230aaa26ec682b632de`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:29.505893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:29.534264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97acd4a-2b43-402d-9bb2-66fa402abc13 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.540308 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.540936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.541213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.541482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.542285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.543960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.545679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:29.556196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1081/2184] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:29.560173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.572443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d079b-65d7-4e84-8d3c-dc893d6b6e66 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.577553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.578224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.578507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.578784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.579434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.581039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.591110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1104/2185] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:29 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:29.593481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1082/2186] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:29.597934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.605017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.607591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:29.608891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102028bb-f001-4de7-87c6-36f38c59a177 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.614185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.614859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.615150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.615428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] PATH_INFO: `/v3/projects/dfd1aa1524cc42e69b3c035f1eb34fac/users/3a6a734533004f15a67a5e5522358d2f/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.631761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfd1aa1524cc42e69b3c035f1eb34fac, user_id=3a6a734533004f15a67a5e5522358d2f, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.634575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb06a3-cf07-401e-9bf0-3de30de62ae0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.657234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1105/2187] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:29 2022] PUT /identity/v3/projects/dfd1aa1524cc42e69b3c035f1eb34fac/users/3a6a734533004f15a67a5e5522358d2f/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:29.667574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.667861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.668134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.680347 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb683bd-e696-4544-abb9-4b5a0a4acc9f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.696271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:29.701434 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] MFA Rules not processed for user `3a6a734533004f15a67a5e5522358d2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:29.702808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:29.744383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.745160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.745506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.745851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.781285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1106/2188] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:29.792477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:29.794299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:29.806940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ce7b1a-1d97-48c0-91fb-bc00071e63f6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:29.810886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.811025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:29.811549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:29.811824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:29.812094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:29.812330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.821263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:29.822578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1083/2189] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:29.822824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88398f3-fbe3-4ab0-a8a7-bdb31f9a06e5 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:29.832399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1107/2190] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:31.828170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:31.966023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbf7df-1b5b-4239-92d2-761f51fa30e8 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:31.977902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:31.978812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:31.979190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:31.980188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] PATH_INFO: `/v3/users/d6df4ccfa3684404aa833b374f38b1d4` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:31.980903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6df4ccfa3684404aa833b374f38b1d4)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:31.983059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:32.038076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6df4ccfa3684404aa833b374f38b1d4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.038670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6df4ccfa3684404aa833b374f38b1d4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.046878 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6df4ccfa3684404aa833b374f38b1d4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.047775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6df4ccfa3684404aa833b374f38b1d4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.055988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6df4ccfa3684404aa833b374f38b1d4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.062022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:32.079914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1084/2191] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:31 2022] DELETE /identity/v3/users/d6df4ccfa3684404aa833b374f38b1d4 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:32.085142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:32.085726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:32.085726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:32.105645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:32.107041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:32.166940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1108/2192] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:32.176813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:32.178379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:32.244725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535fcecc-c6e2-42ee-97ee-692b57c75b1b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:32.248052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:32.249837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:32.249837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:32.249837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:32.259966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:32.261501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:32.279438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1085/2193] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:32.649599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:32.651332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:32.680811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf9905-abf9-49d1-8f49-52517c467aca None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:32.688318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:32.689579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:32.691377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:32.691880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] PATH_INFO: `/v3/projects/7c6b8983a0cb4ccebf0ad1c5ab51e4f9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:32.693419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c6b8983a0cb4ccebf0ad1c5ab51e4f9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:32.696936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:32.716907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:32.773012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c6b8983a0cb4ccebf0ad1c5ab51e4f9', 'request_id': 'req-bd429f85-43fe-4024-9a53-cfce462ea00d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.781371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c6b8983a0cb4ccebf0ad1c5ab51e4f9', 'request_id': 'req-bd429f85-43fe-4024-9a53-cfce462ea00d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.795087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c6b8983a0cb4ccebf0ad1c5ab51e4f9', 'request_id': 'req-bd429f85-43fe-4024-9a53-cfce462ea00d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.815926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] The token cache is being invalidate because project 7c6b8983a0cb4ccebf0ad1c5ab51e4f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:32.816245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.816917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:32.819364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1109/2194] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:32 2022] DELETE /identity/v3/projects/7c6b8983a0cb4ccebf0ad1c5ab51e4f9 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:32.823528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:32.836097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c329552-df7e-425b-814b-5282ab469493 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:32.943136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:32.944430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:32.944715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:32.944994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] PATH_INFO: `/v3/users/e4226354d2f94230aaa26ec682b632de` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:32.955090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4226354d2f94230aaa26ec682b632de)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:32.957152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:33.058061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4226354d2f94230aaa26ec682b632de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.058838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4226354d2f94230aaa26ec682b632de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.078213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4226354d2f94230aaa26ec682b632de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.079900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4226354d2f94230aaa26ec682b632de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.090575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0578ec9-5c5f-4590-bfa5-973fc20750ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4226354d2f94230aaa26ec682b632de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.133917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1086/2195] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:32 2022] DELETE /identity/v3/users/e4226354d2f94230aaa26ec682b632de => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:33.494132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:33.496125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:33.509428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd429f85-43fe-4024-9a53-cfce462ea00d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:33.589909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:33.590888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:33.591175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:33.591456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] PATH_INFO: `/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:33.592906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] RBAC: Authorizing `identity:delete_project(project_id=10f6d3c0a0ca4c18989e9c2e04e6a992)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:33.595412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:33.664244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10f6d3c0a0ca4c18989e9c2e04e6a992', 'request_id': 'req-86c68b4c-416b-4467-9557-9953321d5550'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.672436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10f6d3c0a0ca4c18989e9c2e04e6a992', 'request_id': 'req-86c68b4c-416b-4467-9557-9953321d5550'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.678267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10f6d3c0a0ca4c18989e9c2e04e6a992', 'request_id': 'req-86c68b4c-416b-4467-9557-9953321d5550'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.696406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] The token cache is being invalidate because project 10f6d3c0a0ca4c18989e9c2e04e6a992 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:33.696960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.697842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-86c68b4c-416b-4467-9557-9953321d5550 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:33.700317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1110/2196] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:33 2022] DELETE /identity/v3/projects/10f6d3c0a0ca4c18989e9c2e04e6a992 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:33.706152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:33.706528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:33.706807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:33.726045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:33.734512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:33.736163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:33.831945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1087/2197] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:33.837130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:33.837561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:33.837703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:33.874864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:33.876111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:33.886931 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1111/2198] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:33.891473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:33.903138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3876d903-92f0-44c7-b6fb-7a1b315733d2 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:33.908590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:33.909271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:33.909556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:33.909839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:33.910731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:33.911033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:33.912941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:33.927298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1088/2199] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:33.931423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:33.946497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe42c3b-e4bd-4ace-b4b5-c5c358129b8b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:33.952939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:33.954703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:33.955161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:33.955573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:33.956526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:33.960510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16aba9d-c8f8-403d-960d-6cf18f9ad1d9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.012166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1112/2200] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:33 2022] POST /identity/v3/projects => generated 369 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:34.016658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.046242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8de8d1-29f8-45fc-a50a-e7b47687b42e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.171939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3cee8-b7c1-45cb-a4a3-c03cb759c12f None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.172406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3cee8-b7c1-45cb-a4a3-c03cb759c12f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.172617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a3cee8-b7c1-45cb-a4a3-c03cb759c12f None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.174187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1113/2201] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:03:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:03:34.183938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.184105 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.184421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.204217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:34.224536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.224753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:34.226508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.226637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.227242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.228235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.229087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.232482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.278338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1089/2202] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/users => generated 285 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:34.282895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.295874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4eee9-a395-4b62-a684-df5a88687bac admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.302668 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.304107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.306034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.307329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.308257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.310930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.325246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1090/2203] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:34.330496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.341412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5041bb-434a-4c0a-a872-c898de10a6b0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.345315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1114/2204] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:34.346580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.347648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.348028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.348525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.348809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.349085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.349903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.357300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad717625-4e84-49b2-931f-1c48185d40f0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.364608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd6865-3b31-40d5-ad68-148151698616 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.368724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1091/2205] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:34.371236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9922a53d-a640-4e91-87a8-d7bbbc1559a6 None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.371741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9922a53d-a640-4e91-87a8-d7bbbc1559a6 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.371922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9922a53d-a640-4e91-87a8-d7bbbc1559a6 None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.373389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.373931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1092/2206] 10.222.0.46 () {54 vars in 961 bytes} [Fri Oct 28 11:03:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:03:34.374086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.374343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.374616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] PATH_INFO: `/v3/users/1903d1aaa3444181ac7b63234f4e8970` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.374733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9922a53d-a640-4e91-87a8-d7bbbc1559a6 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.376121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1903d1aaa3444181ac7b63234f4e8970)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.378419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.394686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9922a53d-a640-4e91-87a8-d7bbbc1559a6 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.401248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.402973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.402973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.402973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] PATH_INFO: `/v3/projects/ecd25165e20f477c8d1b40eb7d602e8e/users/1f72b4ab7aed48dc93b2aec728b42315/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.421353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecd25165e20f477c8d1b40eb7d602e8e, user_id=1f72b4ab7aed48dc93b2aec728b42315, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.424191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a3a340-9bac-4352-9e76-a76620be2c3d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.432454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1903d1aaa3444181ac7b63234f4e8970'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.438532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1093/2207] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:34 2022] PUT /identity/v3/projects/ecd25165e20f477c8d1b40eb7d602e8e/users/1f72b4ab7aed48dc93b2aec728b42315/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:34.440027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.440333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.440646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.451714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1903d1aaa3444181ac7b63234f4e8970'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.452463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1903d1aaa3444181ac7b63234f4e8970'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.453061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1903d1aaa3444181ac7b63234f4e8970'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.461377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] MFA Rules not processed for user `73aefca6b8d04a0e8bd2fad4ffedbccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:34.462627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.465845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f84fd319-e5d2-43b6-be17-50dff2f79b34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1903d1aaa3444181ac7b63234f4e8970'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.485792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1115/2208] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:34 2022] DELETE /identity/v3/users/1903d1aaa3444181ac7b63234f4e8970 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:34.490273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.490406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.490667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.498495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:34.520794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] MFA Rules not processed for user `1f72b4ab7aed48dc93b2aec728b42315`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:34.521546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.537034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1094/2209] 10.222.0.46 () {58 vars in 1058 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:34.547155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.559515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42484e05-74d4-4056-9467-1f1d6166e8b1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.592069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1116/2210] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:34.596348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.597659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.604550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:34.632213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.632984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.633683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.633683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.635624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.649474 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.651211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.673502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1095/2211] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:34.674440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.676032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.689322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1dd080-4336-456d-bd05-b037bb961dc8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.692565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242f29ac-9228-4c95-93d3-20f5dd445e9b service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.696520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.696906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.697248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.697578 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.698279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.700305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.701867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.702639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.703053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.703365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] PATH_INFO: `/v3/projects/115fe2408cba49bb8b84af24fc8ce604` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.704541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] RBAC: Authorizing `identity:delete_project(project_id=115fe2408cba49bb8b84af24fc8ce604)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.707163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.732409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1117/2212] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/projects => generated 369 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:34.736649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.756810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '115fe2408cba49bb8b84af24fc8ce604', 'request_id': 'req-aa954803-c7fd-4bd1-b404-d3db3e3cb672'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.762288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '115fe2408cba49bb8b84af24fc8ce604', 'request_id': 'req-aa954803-c7fd-4bd1-b404-d3db3e3cb672'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.768410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '115fe2408cba49bb8b84af24fc8ce604', 'request_id': 'req-aa954803-c7fd-4bd1-b404-d3db3e3cb672'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.782625 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] The token cache is being invalidate because project 115fe2408cba49bb8b84af24fc8ce604 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:34.782910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.783564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aa954803-c7fd-4bd1-b404-d3db3e3cb672 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:34.785055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1096/2213] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:34 2022] DELETE /identity/v3/projects/115fe2408cba49bb8b84af24fc8ce604 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:34.789827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c004-0d31-42ab-bc11-5e78b2269038 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.790133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c004-0d31-42ab-bc11-5e78b2269038 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.790407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447c004-0d31-42ab-bc11-5e78b2269038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.799972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696d45e-10d4-4756-83bf-3837895929c7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.810074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2447c004-0d31-42ab-bc11-5e78b2269038 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:34.811258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2447c004-0d31-42ab-bc11-5e78b2269038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.856389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:34.858007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.858799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.859338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.859606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.860378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.862321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.878009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1097/2214] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:34.884230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1118/2215] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/users => generated 291 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:34.887959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.888462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.888772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.889098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.889687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.903185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee91595-6a44-484b-a660-d5fb0fe3cb5f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.908178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.908848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.909126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.909392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:34.909869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.910086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.910618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:34.911659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.918981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1098/2216] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:34.924731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1119/2217] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:34.924890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.925610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.933317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dec0500-9877-410a-86d2-dad8679359d1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.935059 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef0add-cd94-46e4-81b4-9453e20aeae0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.937843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.938466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.938742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.939117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.939533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.939754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:34.940052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.940147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.940422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.940694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] PATH_INFO: `/v3/projects/096f125be8f749dca594fb87b72a1de6/users/5217d470c52b4d0c9565c53ec2fb03dd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.941624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.953044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:34.953963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1099/2218] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:34.958163 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:34.960562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=096f125be8f749dca594fb87b72a1de6, user_id=5217d470c52b4d0c9565c53ec2fb03dd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.963336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.968467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b04b6-0df8-46b7-a2e0-28f452353439 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:34.973203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:34.973798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:34.974106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:34.974377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:34.975086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:34.976326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1120/2219] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:34 2022] PUT /identity/v3/projects/096f125be8f749dca594fb87b72a1de6/users/5217d470c52b4d0c9565c53ec2fb03dd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:34.976952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:34.980274 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.006050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1100/2220] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:34 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:35.009708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.045189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe656d-d91a-4d74-8451-4024c3d00fbb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.070706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de80feff-35d9-4f5a-8936-8089a1c8a561 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.075465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.076071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.076349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.076616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.077239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.079165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.103444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1101/2221] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:35.104591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.105056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.105676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.105960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.106275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.106976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.108518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.114442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae275cd-a790-41cc-8839-8b575790750d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.120087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1121/2222] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:34 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:35.122883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.123493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.123929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.124204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.124317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.124955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.127734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.135917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6296dd-a7cb-4158-ac1f-1a11a9df14ef admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.141563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1102/2223] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:35.142584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.144692 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.145421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.145627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.145905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] PATH_INFO: `/v3/projects/096f125be8f749dca594fb87b72a1de6/users/5217d470c52b4d0c9565c53ec2fb03dd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.147941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] RBAC: Authorizing `identity:create_grant(project_id=096f125be8f749dca594fb87b72a1de6, user_id=5217d470c52b4d0c9565c53ec2fb03dd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.151756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61627323-a030-421e-b907-c3c47c7ff42b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.153213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9495555-7bbf-475c-b70f-a1e5fb4e405e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.162102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.163951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.171475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.171613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1122/2224] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:35 2022] PUT /identity/v3/projects/096f125be8f749dca594fb87b72a1de6/users/5217d470c52b4d0c9565c53ec2fb03dd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:35.171744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.171744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.171744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.179144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.179455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.180927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.183476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1103/2225] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:35.187832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.201767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0516037-52a4-4c30-8548-49f486a5a8cd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.210442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] MFA Rules not processed for user `5217d470c52b4d0c9565c53ec2fb03dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:35.211694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:35.275838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.276939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.277225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.277498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] PATH_INFO: `/v3/projects/17980fb307ca483f9cae751ebfc36802/users/e856577384cf491e961dca1bdb73807b/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.296843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17980fb307ca483f9cae751ebfc36802, user_id=e856577384cf491e961dca1bdb73807b, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.299916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6aa197-3e13-4ff3-91fe-ea330e96ad73 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.306739 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:35.312974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1123/2226] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:35.328428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1104/2227] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:35 2022] PUT /identity/v3/projects/17980fb307ca483f9cae751ebfc36802/users/e856577384cf491e961dca1bdb73807b/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:35.329751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.331325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:35.339849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.340302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.340579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.343442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:35.370075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] MFA Rules not processed for user `e856577384cf491e961dca1bdb73807b`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:35.371689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:35.394597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a1ee9e-0e6f-4519-95af-52a7f209cc2f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.436476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1105/2228] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:35.440580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.442860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.443477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.443793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.444065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.508711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.509713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff03c20-1dcb-4561-8344-1d9359fbb47f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.510210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.514576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.515214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.515494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.515959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.516621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.518630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.530166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1124/2229] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:35.551832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1106/2230] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:35.557240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.569659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93258700-0509-4a9d-b966-dc383059dc0b service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.570259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.572156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:35.637530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc3dc2a-de15-4884-9b50-69e4de1c9d0f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.640249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.640876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.641150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.641417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.641512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.642076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.642250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.642598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.642940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.644101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.645111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:35.669848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1125/2231] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:35.673546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.678755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:35.683334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:35.691616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6715647f-74bd-44c1-9aad-f9d77a3ea317 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.697111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.697732 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.698160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.698441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.699305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.700942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.716593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1126/2232] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:35.720420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.720563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.721919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.731719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1107/2233] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:35.733551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83272bf-7728-4b23-a164-dcb1bee2af26 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.738749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.739419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.739744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.740007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] PATH_INFO: `/v3/projects/ba879567a5ec417cb8d0e6786d0471bd/users/abc64e4f37ad46169bb513d66c856b85/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.756183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba879567a5ec417cb8d0e6786d0471bd, user_id=abc64e4f37ad46169bb513d66c856b85, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.758875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.775987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1127/2234] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:35 2022] PUT /identity/v3/projects/ba879567a5ec417cb8d0e6786d0471bd/users/abc64e4f37ad46169bb513d66c856b85/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:35.783960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.800083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f22cc-724d-47bd-b56c-7b4b6d3e75bb service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.860394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.861008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.861847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.862328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.862982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8131b824-e627-490e-8423-ffe80e08b976 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.878559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1108/2235] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:35 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:35.880077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:35.891202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc8051-679f-462a-9821-89aa3d4a668c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:35.897853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:35.898680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.899035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.899379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] PATH_INFO: `/v3/projects/ba879567a5ec417cb8d0e6786d0471bd/users/abc64e4f37ad46169bb513d66c856b85/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.901469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba879567a5ec417cb8d0e6786d0471bd, user_id=abc64e4f37ad46169bb513d66c856b85, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:35.904247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:35.915681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1128/2236] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:35 2022] PUT /identity/v3/projects/ba879567a5ec417cb8d0e6786d0471bd/users/abc64e4f37ad46169bb513d66c856b85/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:35.926819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:35.927278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:35.928010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:35.960194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] MFA Rules not processed for user `abc64e4f37ad46169bb513d66c856b85`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:35.961532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:36.054672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1109/2237] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:35 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:36.066949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:36.123280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edefdb55-eb8f-4c76-b7e9-9dbd0d4deca0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:36.127048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:36.127651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:36.127929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:36.128200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:36.137521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:36.138963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9d662a-2762-4f05-9fc0-5c1fc4eff7a7 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:36.148826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1129/2238] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:36 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:36.547788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:36.595049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:36.610728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db6139c-4287-4125-bf23-fb6e4cd8a5bf None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:36.615190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:36.615839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:36.616118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:36.616392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] PATH_INFO: `/v3/users/e856577384cf491e961dca1bdb73807b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:36.617500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e856577384cf491e961dca1bdb73807b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:36.619455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:36.664256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e856577384cf491e961dca1bdb73807b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:36.664821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e856577384cf491e961dca1bdb73807b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:36.672017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e856577384cf491e961dca1bdb73807b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:36.672661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e856577384cf491e961dca1bdb73807b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:36.679824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e856577384cf491e961dca1bdb73807b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:36.696816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1110/2239] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:36 2022] DELETE /identity/v3/users/e856577384cf491e961dca1bdb73807b => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:36.701548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:36.702052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:36.702183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:36.721281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:36.722517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:36.780097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1130/2240] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:36.790313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:36.791691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:36.849044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02419d09-50ca-4dc9-9884-ca4eeb5cff50 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:36.852908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:36.853517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:36.853796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:36.854100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:36.863720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:36.865156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:36.874764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1111/2241] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:37.208626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:37.210998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:37.228088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaa612-ca4d-48d9-b6a8-27be5b279a67 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:37.236007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:37.236939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:37.238026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:37.238026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] PATH_INFO: `/v3/projects/17980fb307ca483f9cae751ebfc36802` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:37.238895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17980fb307ca483f9cae751ebfc36802)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:37.241583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:37.304790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17980fb307ca483f9cae751ebfc36802', 'request_id': 'req-6e8ec8f4-d445-466e-801b-51c0ee4b2555'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.307323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:37.312921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17980fb307ca483f9cae751ebfc36802', 'request_id': 'req-6e8ec8f4-d445-466e-801b-51c0ee4b2555'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.323613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17980fb307ca483f9cae751ebfc36802', 'request_id': 'req-6e8ec8f4-d445-466e-801b-51c0ee4b2555'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.340309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] The token cache is being invalidate because project 17980fb307ca483f9cae751ebfc36802 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:37.340656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.341539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.343225 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1131/2242] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:37 2022] DELETE /identity/v3/projects/17980fb307ca483f9cae751ebfc36802 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:37.347603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:37.360890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c685532b-d95b-4d3f-91a8-ffce9c31a8ce service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:37.435901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:37.436549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:37.436830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:37.437102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] PATH_INFO: `/v3/users/abc64e4f37ad46169bb513d66c856b85` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:37.438888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=abc64e4f37ad46169bb513d66c856b85)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:37.440193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:37.454335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:37.504663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abc64e4f37ad46169bb513d66c856b85'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.505324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abc64e4f37ad46169bb513d66c856b85'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.514472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abc64e4f37ad46169bb513d66c856b85'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.515147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abc64e4f37ad46169bb513d66c856b85'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.523411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7403ebad-f4a6-443f-8096-722c3be95f2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abc64e4f37ad46169bb513d66c856b85'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.542421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1112/2243] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:37 2022] DELETE /identity/v3/users/abc64e4f37ad46169bb513d66c856b85 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:37.815204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:37.816628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:37.835759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ec8f4-d445-466e-801b-51c0ee4b2555 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:37.892232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:37.892904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:37.893181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:37.893453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] PATH_INFO: `/v3/projects/ba879567a5ec417cb8d0e6786d0471bd` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:37.894571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba879567a5ec417cb8d0e6786d0471bd)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:37.896464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:37.941684 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba879567a5ec417cb8d0e6786d0471bd', 'request_id': 'req-4822727d-6dff-4167-8e25-f4a605275a52'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.947246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba879567a5ec417cb8d0e6786d0471bd', 'request_id': 'req-4822727d-6dff-4167-8e25-f4a605275a52'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.953349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba879567a5ec417cb8d0e6786d0471bd', 'request_id': 'req-4822727d-6dff-4167-8e25-f4a605275a52'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.965848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] The token cache is being invalidate because project ba879567a5ec417cb8d0e6786d0471bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:37.966169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.966901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4822727d-6dff-4167-8e25-f4a605275a52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:37.968360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1132/2244] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:37 2022] DELETE /identity/v3/projects/ba879567a5ec417cb8d0e6786d0471bd => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:37.973197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:37.973502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:37.973774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:37.993698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:37.994916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:38.055200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1113/2245] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:38.062820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.063107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.063383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.083149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:38.084919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:38.096389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1133/2246] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:38.097261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.110017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40f429e-2a2a-4e74-b078-77c5189f73fa None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.115115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.115784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.116064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.116334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.116985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:38.117293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.118963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.131627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1114/2247] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:38.137395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.147121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f4553-60c1-4d97-ac73-543300a48ed1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.151774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.152378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.152654 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.152929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.153565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.155535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.186722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1134/2248] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:38.188446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.198607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff38f0b-f792-4c2f-9fde-1799e8965930 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.255164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.255771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.256078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.256348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.257012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.258966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.280756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1115/2249] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:38.284844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.295806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812c82a7-8f16-4d6f-82c7-256ef61391a9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.300365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.300963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.301241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.301512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.302198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.303928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.316707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1135/2250] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:38.321198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.327629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:38.332484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616601b-06ef-4e16-be05-faf8b2bc7db7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.339801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.340408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.340681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.340948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.341590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.343207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa9df63-65aa-485b-9d79-32961d6277c9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.355143 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1116/2251] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:38.359440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.369650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67f6cc-ba99-4b07-afae-e736145d8762 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.374698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.375436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.375773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.376118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] PATH_INFO: `/v3/projects/2baae8ab38be48d7981d6efba43b62b4/users/ec589dba4ddb4be5a340256cecf288bd/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.393168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2baae8ab38be48d7981d6efba43b62b4, user_id=ec589dba4ddb4be5a340256cecf288bd, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.396114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.412969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1136/2252] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:38 2022] PUT /identity/v3/projects/2baae8ab38be48d7981d6efba43b62b4/users/ec589dba4ddb4be5a340256cecf288bd/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:38.435273 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.435650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.435826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.478349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] MFA Rules not processed for user `ec589dba4ddb4be5a340256cecf288bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:38.480183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:38.569051 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1117/2253] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:38.573567 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.586724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:38.660788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efdc524-9252-45d7-bbe1-fdbcef58d7f7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.667442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.668304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.668304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.668858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.669745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.671920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.708683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1137/2254] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/projects => generated 387 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:38.712267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.727401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89678046-4a7a-4f4c-a0cb-6c5e23667465 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.835852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.836652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.837002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.837341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.838263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.840865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.872272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1118/2255] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:38 2022] POST /identity/v3/users => generated 300 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:38.894330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.901656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28d321-140b-4951-9001-067c9bb1c436 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.912002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.912728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.913166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.913551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.914325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.916232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:38.937326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1138/2256] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:38.941368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:38.953491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a4488-bed5-44a5-8901-ac0e7319a771 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:38.962187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:38.963062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:38.963475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:38.963815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] PATH_INFO: `/v3/projects/9280cb5258ec4308802356470c071aba/users/97a586aecdf24af3814bd548e64209b6/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:38.989188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9280cb5258ec4308802356470c071aba, user_id=97a586aecdf24af3814bd548e64209b6, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:38.992661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.008381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1119/2257] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:38 2022] PUT /identity/v3/projects/9280cb5258ec4308802356470c071aba/users/97a586aecdf24af3814bd548e64209b6/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:39.012307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.023343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce044066-2b66-4e2a-94ab-4341d39ce23b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.080960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.081582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.081858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.082179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.082969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.084882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603b79c8-8aad-4fc9-bc8a-77ac4342c1bb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.096226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1139/2258] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:39.101501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.113023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c48153-35cd-4296-9305-1f3b49f00b72 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.118417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.119040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.119218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.119491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] PATH_INFO: `/v3/projects/9280cb5258ec4308802356470c071aba/users/97a586aecdf24af3814bd548e64209b6/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.121370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9280cb5258ec4308802356470c071aba, user_id=97a586aecdf24af3814bd548e64209b6, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.124006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.134450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1120/2259] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:39 2022] PUT /identity/v3/projects/9280cb5258ec4308802356470c071aba/users/97a586aecdf24af3814bd548e64209b6/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:39.144505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.144841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.145120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.173474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] MFA Rules not processed for user `97a586aecdf24af3814bd548e64209b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:39.174755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.257750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1140/2260] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:39 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:39.271340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.273032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.318962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:39.340136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084f10ac-d0ac-4305-a0dd-15ecce610420 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.347876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.349012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.349012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.349012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.359810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.361252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.373230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1121/2261] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:39 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:39.401773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.405966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:39.442705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.444118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.474672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4dac6c-e956-46d0-8ea0-ffa074c99d0d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.481424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.481424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.481880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.482269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] PATH_INFO: `/v3/users/ec589dba4ddb4be5a340256cecf288bd` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.483484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec589dba4ddb4be5a340256cecf288bd)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.485652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.510931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2d011-bbe5-4c5d-9c88-3f512f417825 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.516079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.516751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.517028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.517391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] PATH_INFO: `/v3/users/1f72b4ab7aed48dc93b2aec728b42315` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.518554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f72b4ab7aed48dc93b2aec728b42315)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.520480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.536550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec589dba4ddb4be5a340256cecf288bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.544830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec589dba4ddb4be5a340256cecf288bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.545454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec589dba4ddb4be5a340256cecf288bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.546083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec589dba4ddb4be5a340256cecf288bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.554324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8b6782b3-04aa-4b81-98e3-16b9b6759600 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec589dba4ddb4be5a340256cecf288bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.574788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1141/2262] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:39 2022] DELETE /identity/v3/users/ec589dba4ddb4be5a340256cecf288bd => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:39.579111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.579463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.579745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.588829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f72b4ab7aed48dc93b2aec728b42315'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.589634 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f72b4ab7aed48dc93b2aec728b42315'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.598262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f72b4ab7aed48dc93b2aec728b42315'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.598894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f72b4ab7aed48dc93b2aec728b42315'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.599534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:39.601455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.611086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c4851091-ca97-4980-93c9-eefbf81c83d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f72b4ab7aed48dc93b2aec728b42315'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:39.633139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1122/2263] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:39 2022] DELETE /identity/v3/users/1f72b4ab7aed48dc93b2aec728b42315 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:39.638272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.638660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.638998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.671680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:39.671900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1142/2264] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:39.673209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.682456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.684454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.722048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:39.732034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:39.745169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1123/2265] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:39.754465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:39.755917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:39.765923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38a1a3-261e-40e3-a9d3-6ea462ea0178 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.812846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261e682-60c0-4170-a45d-8b2a5dbb12a7 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:39.814619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.815270 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.815548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.815820 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.816604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:39.817229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:39.817487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:39.817758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:39.825416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.826908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.827152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:39.828607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:39.837731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1143/2266] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:39.839623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1124/2267] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:40.221109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:40.224219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:40.242626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88d4e7-3b85-454f-a582-07435c634936 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:40.259852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:40.261532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:40.278353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:40.279167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:40.280704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:40.280971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] PATH_INFO: `/v3/projects/2baae8ab38be48d7981d6efba43b62b4` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:40.283680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2baae8ab38be48d7981d6efba43b62b4)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:40.285761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:40.290153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6a0f66-40f7-4272-9f9b-82e1c5b9aaed service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:40.300891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:40.301576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:40.301864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:40.302229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] PATH_INFO: `/v3/projects/ecd25165e20f477c8d1b40eb7d602e8e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:40.304642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ecd25165e20f477c8d1b40eb7d602e8e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:40.306734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:40.344587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2baae8ab38be48d7981d6efba43b62b4', 'request_id': 'req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.350889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2baae8ab38be48d7981d6efba43b62b4', 'request_id': 'req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.366612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2baae8ab38be48d7981d6efba43b62b4', 'request_id': 'req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.366719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ecd25165e20f477c8d1b40eb7d602e8e', 'request_id': 'req-b2c541e3-9103-4e47-9eda-969991c7f63d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.373282 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ecd25165e20f477c8d1b40eb7d602e8e', 'request_id': 'req-b2c541e3-9103-4e47-9eda-969991c7f63d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.379951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] The token cache is being invalidate because project 2baae8ab38be48d7981d6efba43b62b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:40.379951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.380148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ecd25165e20f477c8d1b40eb7d602e8e', 'request_id': 'req-b2c541e3-9103-4e47-9eda-969991c7f63d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.380764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.383195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1144/2268] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:40 2022] DELETE /identity/v3/projects/2baae8ab38be48d7981d6efba43b62b4 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:40.387372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:40.389129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:40.395694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] The token cache is being invalidate because project ecd25165e20f477c8d1b40eb7d602e8e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:40.396248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.397077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.398989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1125/2269] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:40 2022] DELETE /identity/v3/projects/ecd25165e20f477c8d1b40eb7d602e8e => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:40.410822 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:40.417904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:40.516391 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc3ad3-70ec-45c3-86e3-40f783b2793b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:40.517532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c541e3-9103-4e47-9eda-969991c7f63d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:40.523819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:40.524361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:40.524713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:40.525076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:40.525195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:40.525341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:40.525614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] PATH_INFO: `/v3/users/5217d470c52b4d0c9565c53ec2fb03dd` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:40.525934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] PATH_INFO: `/v3/users/97a586aecdf24af3814bd548e64209b6` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:40.526962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5217d470c52b4d0c9565c53ec2fb03dd)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:40.527561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97a586aecdf24af3814bd548e64209b6)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:40.529611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:40.530963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:40.540023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:40.568252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:40.605755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5217d470c52b4d0c9565c53ec2fb03dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.606598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5217d470c52b4d0c9565c53ec2fb03dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.618859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5217d470c52b4d0c9565c53ec2fb03dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.620654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5217d470c52b4d0c9565c53ec2fb03dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.624788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97a586aecdf24af3814bd548e64209b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.631860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5217d470c52b4d0c9565c53ec2fb03dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.635380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97a586aecdf24af3814bd548e64209b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.636236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97a586aecdf24af3814bd548e64209b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.637638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97a586aecdf24af3814bd548e64209b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.654087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97a586aecdf24af3814bd548e64209b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:40.662801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1126/2270] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:40 2022] DELETE /identity/v3/users/5217d470c52b4d0c9565c53ec2fb03dd => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:40.683766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1145/2271] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:40 2022] DELETE /identity/v3/users/97a586aecdf24af3814bd548e64209b6 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:41.026199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.027461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.107135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.108941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.121568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b209468-dd82-4642-9a99-7d670418b892 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.203490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a258ad5a-8a0a-4fb2-9c60-28b90ee59437 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.210207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.211074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.211362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] PATH_INFO: `/v3/projects/9280cb5258ec4308802356470c071aba` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.212871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9280cb5258ec4308802356470c071aba)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.215065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.255876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.256513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.256785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.257044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] PATH_INFO: `/v3/projects/096f125be8f749dca594fb87b72a1de6` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.258251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=096f125be8f749dca594fb87b72a1de6)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.260356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.297380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9280cb5258ec4308802356470c071aba', 'request_id': 'req-aa6811ff-5ab9-489b-b616-a8bced897010'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.312830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9280cb5258ec4308802356470c071aba', 'request_id': 'req-aa6811ff-5ab9-489b-b616-a8bced897010'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.324362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9280cb5258ec4308802356470c071aba', 'request_id': 'req-aa6811ff-5ab9-489b-b616-a8bced897010'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.338288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '096f125be8f749dca594fb87b72a1de6', 'request_id': 'req-d538265a-0ab3-421c-8fcc-819700ef5e3e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.340430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] The token cache is being invalidate because project 9280cb5258ec4308802356470c071aba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:41.340662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.341564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-aa6811ff-5ab9-489b-b616-a8bced897010 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.342133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:41.343760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1146/2272] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:41 2022] DELETE /identity/v3/projects/9280cb5258ec4308802356470c071aba => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:41.348514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd68736-8466-42b7-85a9-a84a3622b980 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.348760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd68736-8466-42b7-85a9-a84a3622b980 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.349044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd68736-8466-42b7-85a9-a84a3622b980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.351826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '096f125be8f749dca594fb87b72a1de6', 'request_id': 'req-d538265a-0ab3-421c-8fcc-819700ef5e3e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.370784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '096f125be8f749dca594fb87b72a1de6', 'request_id': 'req-d538265a-0ab3-421c-8fcc-819700ef5e3e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.374043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-dbd68736-8466-42b7-85a9-a84a3622b980 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:41.375615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dbd68736-8466-42b7-85a9-a84a3622b980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.391938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:41.394519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] The token cache is being invalidate because project 096f125be8f749dca594fb87b72a1de6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:41.394989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.395912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d538265a-0ab3-421c-8fcc-819700ef5e3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:41.399572 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1127/2273] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:41 2022] DELETE /identity/v3/projects/096f125be8f749dca594fb87b72a1de6 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:41.404660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d64af1-e419-42a8-8972-c11fa3db556d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.404981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d64af1-e419-42a8-8972-c11fa3db556d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.405219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d64af1-e419-42a8-8972-c11fa3db556d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.428797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a9d64af1-e419-42a8-8972-c11fa3db556d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:41.430079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a9d64af1-e419-42a8-8972-c11fa3db556d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.477342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1147/2274] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:41.480477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.480892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.481171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.502591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:41.503893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.512943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1148/2275] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:41.516845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.518114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1128/2276] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:41.530359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.530359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.530359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.530810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41a045a-709d-4a47-aa8b-b5771a4448a2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.535427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.536067 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.536513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.536659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.537332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:41.537666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.539561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.549731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:41.551176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:41.551933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1149/2277] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:41.557517 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.564250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1129/2278] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:41.565246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.570742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c5272-5ef7-4b78-9a2d-f8c0ac6d71f0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.576588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.577060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3e62f4-e0cd-4089-a226-9c33aac63afe None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.577359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.577704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.578143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.578779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.580895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.586054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.586680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.586954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.587375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.588131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:41.588429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.590165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.604304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1130/2279] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:41.617276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1150/2280] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/projects => generated 393 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:41.618602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.618740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.638289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fe9345-3b63-46dc-817d-78c2216961ce admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.718292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.718927 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.719329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.719606 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.720674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09084a37-dd34-46d1-9962-42335ef6fe75 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.721148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.723372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.725903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.726639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.727004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.727378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.728090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.730219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.765176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1151/2281] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/users => generated 297 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:41.774534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.783222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1131/2282] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/projects => generated 383 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:41.787818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.791653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0cef9c-abf5-41af-a464-0f5134b8c37a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.848889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:41.867741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d195b8a3-2224-450b-89fc-f736b18b3924 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.870781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.871600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.872071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.872384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.872694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.872995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.873269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.873552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.873675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.874195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.875128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.876096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.877799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:41.889638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1152/2283] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:41.892581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.907493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1132/2284] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:41 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:41.909136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46957e87-7338-47b9-a0f5-33d243c30596 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.911558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.914879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.915565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.916018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.916312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.917136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.918874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.922751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d609e-f0ee-4ad8-88a8-07e87e83f167 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.928649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.929327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.929702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.930102 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.930856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.934190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1153/2285] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:41.934302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.935555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.947083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7938de3e-4997-4e53-9b36-0ced05a139f2 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.947435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1133/2286] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:41.950882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:41.954407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.955156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.955507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.955962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] PATH_INFO: `/v3/projects/d4dd88967cc14d9a9ba110b617a99f40/users/a7556d3670f44a99a2fe8caa1c74e1f3/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.969040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c1e0de-df9a-486f-8098-9dc2da10db44 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:41.973723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4dd88967cc14d9a9ba110b617a99f40, user_id=a7556d3670f44a99a2fe8caa1c74e1f3, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.975704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:41.976327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:41.976610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:41.977028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:41.977682 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:41.978555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19324f3-40b5-4d62-8353-dc16a63f8dea admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.979633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:41.998895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1134/2287] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:41 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:42.003019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.013051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1154/2288] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:41 2022] PUT /identity/v3/projects/d4dd88967cc14d9a9ba110b617a99f40/users/a7556d3670f44a99a2fe8caa1c74e1f3/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:42.016641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9caf6a-3308-4083-9830-1573ba9e014b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.023385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.023680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.023965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.053536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] MFA Rules not processed for user `a7556d3670f44a99a2fe8caa1c74e1f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:42.054845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:42.119570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.120425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.120779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.121121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] PATH_INFO: `/v3/projects/2735a5387cef47149c405e6056ba5b9b/users/0ec732e209d8444ebe8a6a2879d3e39b/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.139605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2735a5387cef47149c405e6056ba5b9b, user_id=0ec732e209d8444ebe8a6a2879d3e39b, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.142767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38901586-eb40-4cf7-98b7-1fec92139306 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.145636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1155/2289] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:42.150427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.160896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1135/2290] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:42 2022] PUT /identity/v3/projects/2735a5387cef47149c405e6056ba5b9b/users/0ec732e209d8444ebe8a6a2879d3e39b/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:42.163122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198f6354-2c88-4e47-8b19-ebe0e03b4aa5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.171396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.171687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.171971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.207576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] MFA Rules not processed for user `0ec732e209d8444ebe8a6a2879d3e39b`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:42.209053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:42.240047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.241712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.242173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.242633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.243396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.245389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.291591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1156/2291] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/projects => generated 391 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:42.295795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.296785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1136/2292] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:42.301064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.313619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975a70f5-e681-4749-b066-883a79268ef5 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.342414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:42.351514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:42.377740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3205eae-c0fa-48f8-9175-8be898444e0a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.383481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.384191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.384659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.384813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.385269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.385424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.385687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.385952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.386078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.386596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.388487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.388607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.420117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1157/2293] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/users => generated 302 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:42.420384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1137/2294] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/projects => generated 385 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:42.422904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.427141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.439584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feecfbf-b5e6-4197-a56c-3816ee1009c7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.535761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.536685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.537854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.539059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b01b6-1e16-4144-a758-b66ea5cdd84b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.539831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.541788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.544318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.544456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.544939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.545218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.545493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.546337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.548235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.573338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1138/2295] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/users => generated 299 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:42.574333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1158/2296] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:42 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:42.579300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.586303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.595758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a697b-b551-476a-8f30-5dd25384257e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.605067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.605412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.605695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.605969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.606813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.606942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3165927-2f0e-4a83-9b25-587e31b2bc55 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.608525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.615673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.616477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.616827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.617206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] PATH_INFO: `/v3/projects/83da2ca7249549ab907f94c1583a27c5/users/aa17b670ae424b10a355786ec92c70ab/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.622446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1139/2297] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:42 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:42.626439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.637276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ea6658-eb85-45c2-b6c2-b795a83dd386 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.641626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83da2ca7249549ab907f94c1583a27c5, user_id=aa17b670ae424b10a355786ec92c70ab, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.644656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.645003 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.645832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.646991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.647276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] PATH_INFO: `/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f/users/2468ceb260864967bc3c6282624a203f/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.666254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1159/2298] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:42 2022] PUT /identity/v3/projects/83da2ca7249549ab907f94c1583a27c5/users/aa17b670ae424b10a355786ec92c70ab/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:42.669966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.670272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5e0f0adbfe3442185a3f1693b70a61f, user_id=2468ceb260864967bc3c6282624a203f, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.673686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.696111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1140/2299] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:42 2022] PUT /identity/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f/users/2468ceb260864967bc3c6282624a203f/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:42.700617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.761356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023a3415-952a-444f-8756-ea745fdae9d7 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.816736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08446108-e0b6-49d8-9cfb-4969f6930400 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.822743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.823508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.823891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.824268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.825107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.826990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.846170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1141/2300] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:42 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:42.846675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.852140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:42.857964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:42.859844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9847ab07-19c6-466e-bbfe-31bd26c3dc1d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.873876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.874594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.875022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.875369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] PATH_INFO: `/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f/users/2468ceb260864967bc3c6282624a203f/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.877069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:42.877828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.878149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5e0f0adbfe3442185a3f1693b70a61f, user_id=2468ceb260864967bc3c6282624a203f, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.878401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.878798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.879520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:42.881877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c97aa3-bf94-4f79-9e01-0deebd5be011 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.882154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:42.896247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1160/2302] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:42 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:42.898215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1142/2302] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:42 2022] PUT /identity/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f/users/2468ceb260864967bc3c6282624a203f/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:42.900669 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:42.911633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:42.911945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:42.912538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:42.923411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e80ba19-8b59-4b46-9f57-e4a9bec466a9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:42.947874 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] MFA Rules not processed for user `2468ceb260864967bc3c6282624a203f`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:42.949943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.038772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.039476 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.039756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.039989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] PATH_INFO: `/v3/projects/83da2ca7249549ab907f94c1583a27c5/users/aa17b670ae424b10a355786ec92c70ab/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.050191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] RBAC: Authorizing `identity:create_grant(project_id=83da2ca7249549ab907f94c1583a27c5, user_id=aa17b670ae424b10a355786ec92c70ab, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.053208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75af6d99-9970-44fa-a213-5bdd8245c28d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.075620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1161/2303] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:42 2022] PUT /identity/v3/projects/83da2ca7249549ab907f94c1583a27c5/users/aa17b670ae424b10a355786ec92c70ab/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:43.076943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1143/2304] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:42 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:43.077083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.081107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.081471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.091413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.093569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.114703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] MFA Rules not processed for user `aa17b670ae424b10a355786ec92c70ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:43.114703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.185460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8069b8d1-1d9f-4f0b-8ebc-809c5a5bddea None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:43.197645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1162/2305] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:43 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:43.230289 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.231633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.239557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.240700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.240984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.241254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.292025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:43.299009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e712173-52c7-4d71-bde4-1a375adc8a03 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:43.304298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.304879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.305166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.305443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] PATH_INFO: `/v3/users/f74566bec8b54837b55822a45a4543e5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.305953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.306638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f74566bec8b54837b55822a45a4543e5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.307416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.308555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.311354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:43.317016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1144/2306] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:43.317914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.319971 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.330483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4284220b-1bf6-4eea-a862-a3cf4cabb07d service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:43.335787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.336628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.336970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.337245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.347768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.349203 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6accba-b0e7-4701-a215-2c9d6ca86d63 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.358989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f74566bec8b54837b55822a45a4543e5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:43.359764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1145/2307] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:43 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:43.371915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f74566bec8b54837b55822a45a4543e5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:43.372743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f74566bec8b54837b55822a45a4543e5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:43.373449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f74566bec8b54837b55822a45a4543e5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:43.382534 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f74566bec8b54837b55822a45a4543e5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:43.401810 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1163/2308] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:43 2022] DELETE /identity/v3/users/f74566bec8b54837b55822a45a4543e5 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:43.412012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.412517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.412694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.433377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:43.434616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.448938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.450508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.517700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1146/2309] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:43.523435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b99e2-a5e7-4f6b-b41e-65790521b679 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:43.527447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.528532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.529477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.529615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.529615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.529759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.531668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:43.597450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7adf3cf-554e-4ebd-b0d5-c1bec1aa580f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:43.601880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:43.602515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:43.602791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:43.603059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:43.610579 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.613083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.613414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:43.614968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c77b335-c73f-4da3-86dd-3e0e207e3938 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:43.623750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1164/2310] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:43 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:43.629223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1147/2311] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:43.975155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:43.976649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:44.012454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da137e8-aa90-44ae-bc85-866bea92bcbb service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.033088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.033763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.034095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.034379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] PATH_INFO: `/v3/projects/1ed949adee1f461db75dfbc192abd945` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.035538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ed949adee1f461db75dfbc192abd945)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.038248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.073368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:44.087756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ed949adee1f461db75dfbc192abd945', 'request_id': 'req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:44.094166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ed949adee1f461db75dfbc192abd945', 'request_id': 'req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:44.101023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ed949adee1f461db75dfbc192abd945', 'request_id': 'req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:44.113168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] The token cache is being invalidate because project 1ed949adee1f461db75dfbc192abd945 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:44.113457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:44.114096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f1c3a1f0-9105-42ad-acd2-07bf08bf27cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:44.115806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1165/2312] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:43 2022] DELETE /identity/v3/projects/1ed949adee1f461db75dfbc192abd945 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:44.120808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.121320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.121724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.140893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:44.142122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:44.204204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1148/2313] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:44.213107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.213389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.213673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.234900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:44.236194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:44.244490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1166/2314] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:44.249517 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.259674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ab8ed8-55d9-40b4-b503-f8f7c3bedb0c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.264781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.265386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.265688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.265959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.266641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:44.266942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.268636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.281298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1149/2315] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:44.285512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.295327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f40d96a-1870-4c8b-adb0-a7acdf698d88 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.300151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.300749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.301024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.301293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.301918 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.303871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.331670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1167/2316] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:44 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:44.336660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.343100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:44.347354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5961fb4c-482a-4524-860d-122d681bed3c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.403070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.403779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.404063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.404338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.404980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.407005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.429225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1150/2317] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:44 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:44.433340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.444135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706be67-6ea6-4728-a002-a0aac0a1c544 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.449046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.449629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.449940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.450217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.450926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.452518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.463714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1168/2318] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:44.467622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.477939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255e8c69-a00e-429f-b044-18248ddf253a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.483728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.483883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.484544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.484821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.485491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.487108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58be932c-72b3-4c31-923c-028d5c7cba52 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.497834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1151/2319] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:44.502011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.512629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad9dcd6-8e24-473f-b243-803735800ebf admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.517199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.517855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.518169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.518514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] PATH_INFO: `/v3/projects/cf9538379ffd44ee9a3765f8dea19810/users/e5a66fd6314e43a28696e7faec4eb1b6/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.532731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf9538379ffd44ee9a3765f8dea19810, user_id=e5a66fd6314e43a28696e7faec4eb1b6, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.535407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.548330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1169/2320] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:44 2022] PUT /identity/v3/projects/cf9538379ffd44ee9a3765f8dea19810/users/e5a66fd6314e43a28696e7faec4eb1b6/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:44.559145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c944a32-f23e-46f3-ab90-e8b5cbac13d8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.559432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c944a32-f23e-46f3-ab90-e8b5cbac13d8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.559708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c944a32-f23e-46f3-ab90-e8b5cbac13d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.586408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-6c944a32-f23e-46f3-ab90-e8b5cbac13d8 None None] MFA Rules not processed for user `e5a66fd6314e43a28696e7faec4eb1b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:44.587605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6c944a32-f23e-46f3-ab90-e8b5cbac13d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:44.650449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1152/2321] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:44.660977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:44.663329 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:44.721868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a532cf9-843c-48f9-ae6f-f730bea5db72 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:44.726360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:44.727081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:44.727422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:44.727796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:44.738096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:44.739734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326ca3ca-07e3-451d-8976-618a625c7b98 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:44.749885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1170/2322] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:47.178936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:47.179470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:47.179470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:47.185278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ede4f-f7c5-4042-9e27-76f7a51c3995 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:47.185717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ede4f-f7c5-4042-9e27-76f7a51c3995 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:47.185812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ede4f-f7c5-4042-9e27-76f7a51c3995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:47.201488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] MFA Rules not processed for user `e5a66fd6314e43a28696e7faec4eb1b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:47.202755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:47.206499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-498ede4f-f7c5-4042-9e27-76f7a51c3995 None None] MFA Rules not processed for user `6b926f7161c0487a8c2b04ada3b374e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:47.207667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-498ede4f-f7c5-4042-9e27-76f7a51c3995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:47.211053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1153/2323] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:47.288589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1171/2324] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:47 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:48.314401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:48.316504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:48.392786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1806889a-6663-40a9-8e2e-184bac8bb57c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:48.397915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:48.398725 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:48.399172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:48.399450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] PATH_INFO: `/v3/users/a7556d3670f44a99a2fe8caa1c74e1f3` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:48.400565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7556d3670f44a99a2fe8caa1c74e1f3)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:48.402516 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:48.407943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:48.454271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7556d3670f44a99a2fe8caa1c74e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:48.455307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7556d3670f44a99a2fe8caa1c74e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:48.463923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7556d3670f44a99a2fe8caa1c74e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:48.464547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7556d3670f44a99a2fe8caa1c74e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:48.472709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7556d3670f44a99a2fe8caa1c74e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:48.494295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1154/2325] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:48 2022] DELETE /identity/v3/users/a7556d3670f44a99a2fe8caa1c74e1f3 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:48.499623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:48.500554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:48.500872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:48.509337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:48.525591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:48.527719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:48.589047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1172/2326] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:48.598772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:48.600916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:48.667772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578feee-89a4-4075-9859-6cf723beb972 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:48.671519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:48.672219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:48.672577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:48.672921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:48.682487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:48.683928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:48.693355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1155/2327] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:49.016976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:49.026191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:49.057076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972d52a6-840f-450f-9182-56d8264761f1 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:49.062216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:49.062872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:49.063207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:49.063496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] PATH_INFO: `/v3/projects/d4dd88967cc14d9a9ba110b617a99f40` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:49.064812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4dd88967cc14d9a9ba110b617a99f40)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:49.066990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:49.156766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4dd88967cc14d9a9ba110b617a99f40', 'request_id': 'req-eb0b32da-dc34-4195-a454-4b4f7fd03947'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.181390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4dd88967cc14d9a9ba110b617a99f40', 'request_id': 'req-eb0b32da-dc34-4195-a454-4b4f7fd03947'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.201668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4dd88967cc14d9a9ba110b617a99f40', 'request_id': 'req-eb0b32da-dc34-4195-a454-4b4f7fd03947'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.219893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:49.221269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:49.223236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] The token cache is being invalidate because project d4dd88967cc14d9a9ba110b617a99f40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:49.223559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.224454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.230056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1173/2328] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:49 2022] DELETE /identity/v3/projects/d4dd88967cc14d9a9ba110b617a99f40 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:49.237280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eac4cc-011d-47fc-8872-1d444987bf58 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:49.238981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:49.240651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:49.331534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:49.332265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:49.332648 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:49.332977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] PATH_INFO: `/v3/users/3a6a734533004f15a67a5e5522358d2f` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:49.335070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a6a734533004f15a67a5e5522358d2f)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:49.337182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:49.343100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b32da-dc34-4195-a454-4b4f7fd03947 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:49.348352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:49.349198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:49.350332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:49.350752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] PATH_INFO: `/v3/users/aa17b670ae424b10a355786ec92c70ab` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:49.352046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa17b670ae424b10a355786ec92c70ab)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:49.354180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:49.394543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:49.418111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:49.432319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a6a734533004f15a67a5e5522358d2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.433400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a6a734533004f15a67a5e5522358d2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.450543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a6a734533004f15a67a5e5522358d2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.451448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a6a734533004f15a67a5e5522358d2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.456933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa17b670ae424b10a355786ec92c70ab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.465954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-70bfe879-c4e2-4b83-8699-c0d9a5d1bb7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a6a734533004f15a67a5e5522358d2f'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.471996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa17b670ae424b10a355786ec92c70ab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.473613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa17b670ae424b10a355786ec92c70ab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.474716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa17b670ae424b10a355786ec92c70ab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.495303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa17b670ae424b10a355786ec92c70ab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:49.499825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1156/2329] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:49 2022] DELETE /identity/v3/users/3a6a734533004f15a67a5e5522358d2f => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:49.504364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:49.504841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:49.505382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:49.520437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1174/2330] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:49 2022] DELETE /identity/v3/users/aa17b670ae424b10a355786ec92c70ab => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:49.527500 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:49.528700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:49.618959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1157/2331] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:49.626924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:49.629395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:49.749925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1b4b6-6c7b-463c-b265-71d2652f6cdb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:49.759902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:49.760634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:49.761170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:49.761454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:49.778845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:49.789672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:49.817640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1175/2332] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:49.983031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:49.984758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.032093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14761b76-9aae-4477-afb8-89d5639ee12d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.060217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.062225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.063793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.064222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] PATH_INFO: `/v3/projects/83da2ca7249549ab907f94c1583a27c5` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.067997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] RBAC: Authorizing `identity:delete_project(project_id=83da2ca7249549ab907f94c1583a27c5)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.073484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.198208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83da2ca7249549ab907f94c1583a27c5', 'request_id': 'req-9f68e716-d791-4f5f-888e-d622fd037a5e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.218788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83da2ca7249549ab907f94c1583a27c5', 'request_id': 'req-9f68e716-d791-4f5f-888e-d622fd037a5e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.242208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83da2ca7249549ab907f94c1583a27c5', 'request_id': 'req-9f68e716-d791-4f5f-888e-d622fd037a5e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.288447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] The token cache is being invalidate because project 83da2ca7249549ab907f94c1583a27c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:50.288890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.289768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9f68e716-d791-4f5f-888e-d622fd037a5e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.291815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.294013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.296564 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1158/2333] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:49 2022] DELETE /identity/v3/projects/83da2ca7249549ab907f94c1583a27c5 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:50.301820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c642f41a-4227-4e76-b55d-420361b50fbc None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.303571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c642f41a-4227-4e76-b55d-420361b50fbc None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.304425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c642f41a-4227-4e76-b55d-420361b50fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.336941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c642f41a-4227-4e76-b55d-420361b50fbc None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:50.338928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c642f41a-4227-4e76-b55d-420361b50fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.376602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:50.383162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb97d1-3d1e-4861-8f72-f41eb75253f2 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.442431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1159/2334] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:50.451526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.452179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.452803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.472660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.473390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.473737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.474130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.476228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.479858 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:50.481443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.495301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1160/2335] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:50.502095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.502999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:50.518762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec029c2-ae71-48d1-9d28-2b3181f1e2b0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.524942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.525850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.526227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.526563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.527888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:50.528266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.530221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.546946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1161/2336] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:50.548696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.564975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108a5f7-a4d0-473f-844d-14e2f2a95697 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.569554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.572218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.577730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.578393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.578674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.578947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.579720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.582191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.589688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1176/2337] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:50 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:50.622101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.623755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:50.647049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1162/2338] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:50 2022] POST /identity/v3/projects => generated 371 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:50.652184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.660022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a810cb-1b46-4439-9ee5-da4478aa7b66 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.748301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.749330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.749800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.750460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] PATH_INFO: `/v3/projects/dfd1aa1524cc42e69b3c035f1eb34fac` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.752111 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfd1aa1524cc42e69b3c035f1eb34fac)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.754911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.792126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b67832d-af75-4261-8c3b-81075b197438 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.798146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:50.799343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:50.799687 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:50.800109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:50.801059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:50.804382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:50.844690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfd1aa1524cc42e69b3c035f1eb34fac', 'request_id': 'req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.852145 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1163/2339] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:50 2022] POST /identity/v3/users => generated 286 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:50.860508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:50.879213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfd1aa1524cc42e69b3c035f1eb34fac', 'request_id': 'req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.887547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfd1aa1524cc42e69b3c035f1eb34fac', 'request_id': 'req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.896444 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3def0d90-1428-42a6-8c6f-71cb3cfcc550 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:50.923142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] The token cache is being invalidate because project dfd1aa1524cc42e69b3c035f1eb34fac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:50.923452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.924296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:50.926524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1177/2340] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:50 2022] DELETE /identity/v3/projects/dfd1aa1524cc42e69b3c035f1eb34fac => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:51.110091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:51.110780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.111152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.111321 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.112109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:51.113896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:51.174435 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1164/2341] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:50 2022] GET /identity/v3/roles => generated 1420 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:51.179093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:51.180602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:51.202175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4b68ec-8ee5-4fc5-b9a5-50859d1b6873 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:51.232208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:51.233090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.233440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.233803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.234806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:51.241664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefdd634-1c78-47b7-a756-013c72face8b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:51.281393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1178/2342] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:51 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:51.284372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:51.307241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f331d0-5451-4910-9e7e-d32c0f941232 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:51.316697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:51.317898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.317898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.318350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] PATH_INFO: `/v3/projects/266ff39a3d7748ddb9f1fdaa92305c5a/users/a5617451445043d3a09629e312d3b661/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.355917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:51.363010 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] RBAC: Authorizing `identity:create_grant(project_id=266ff39a3d7748ddb9f1fdaa92305c5a, user_id=a5617451445043d3a09629e312d3b661, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:51.368534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:51.391386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1165/2343] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:51 2022] PUT /identity/v3/projects/266ff39a3d7748ddb9f1fdaa92305c5a/users/a5617451445043d3a09629e312d3b661/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:51.404124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.404608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.405076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.445357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] MFA Rules not processed for user `a5617451445043d3a09629e312d3b661`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:51.446936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:51.479640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:51.547369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1179/2344] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:51.560111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:51.562956 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:51.640949 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e7636e-1ca0-4376-95a2-760c44e18059 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:51.648520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:51.649190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.649475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.649759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.661838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:51.664016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8af8c4-b496-4153-bb9b-7a1b2c4b5944 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:51.676467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1166/2345] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:51 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:51.703086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:51.778197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76b8a7c-1d6b-4228-a76b-078f46ef5406 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:51.785244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:51.786822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.788229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.788887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] PATH_INFO: `/v3/users/a5617451445043d3a09629e312d3b661` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.790384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5617451445043d3a09629e312d3b661)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:51.792591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:51.857210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5617451445043d3a09629e312d3b661'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:51.869310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5617451445043d3a09629e312d3b661'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:51.870345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5617451445043d3a09629e312d3b661'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:51.871300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5617451445043d3a09629e312d3b661'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:51.892680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5617451445043d3a09629e312d3b661'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:51.923839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1180/2346] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:51 2022] DELETE /identity/v3/users/a5617451445043d3a09629e312d3b661 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:51.929034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:51.929325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:51.929603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:51.959652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:51.960900 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:52.050058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1167/2347] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:52.062788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:52.064760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:52.145058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:52.159995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aba5cf-b0dc-4168-bfd1-c8ca2844efed admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:52.169565 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:52.170601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:52.170910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:52.172022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:52.188610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:52.190528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b48f5d-7a5a-4c68-b5a3-5bb5d181e052 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:52.203101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1181/2348] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:53.430141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:53.431839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:53.447420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79885324-53f6-4db2-bbde-5a6a39412b68 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:53.452718 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:53.453635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.454345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.454831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] PATH_INFO: `/v3/projects/266ff39a3d7748ddb9f1fdaa92305c5a` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.456588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=266ff39a3d7748ddb9f1fdaa92305c5a)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:53.458766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:53.555502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '266ff39a3d7748ddb9f1fdaa92305c5a', 'request_id': 'req-988f135a-51e0-422f-971a-f5f4ae97beb2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:53.566009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '266ff39a3d7748ddb9f1fdaa92305c5a', 'request_id': 'req-988f135a-51e0-422f-971a-f5f4ae97beb2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:53.573846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '266ff39a3d7748ddb9f1fdaa92305c5a', 'request_id': 'req-988f135a-51e0-422f-971a-f5f4ae97beb2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:53.594256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] The token cache is being invalidate because project 266ff39a3d7748ddb9f1fdaa92305c5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:53.595000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:53.596190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-988f135a-51e0-422f-971a-f5f4ae97beb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:53.598187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1168/2349] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:53 2022] DELETE /identity/v3/projects/266ff39a3d7748ddb9f1fdaa92305c5a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:53.604249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.604679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.604845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.635700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:53.637095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:53.719566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1182/2350] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:53.723844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.724135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.725043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.750598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:53.751808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:53.759973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1169/2351] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:53.764084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:53.775336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89fda0e-702f-4a3c-bcbd-de599dbb71fa None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:53.779977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:53.780672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.781021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.781356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.782099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:53.782475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:53.784129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:53.802399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1183/2352] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:53.803751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:53.815693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c5a6c-1a7a-4f26-8298-b686d64dd727 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:53.820570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:53.821267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.821599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.821939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.822666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:53.824643 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:53.831096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:53.862707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1170/2353] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:53 2022] POST /identity/v3/projects => generated 383 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:53.867574 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:53.880474 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224b05ac-b4b5-4f49-85dd-658114037856 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:53.967634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:53.968293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:53.968575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:53.968846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:53.969478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:53.971435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.000978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1184/2354] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:53 2022] POST /identity/v3/users => generated 292 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:54.003012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:54.022681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fece64c-943f-4779-bee9-c1dfc2473ebb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:54.027889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:54.028601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.028962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.029788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.030614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:54.032562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.046796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1171/2355] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:54.050902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:54.063239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c38f8f8-4167-4ed7-8a74-2d2c0373b625 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:54.068969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:54.070019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.070313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.071215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.073451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:54.075744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7c88ca-0215-4993-8e9e-a6373b958a4a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.090192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1185/2356] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:54.096099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:54.109576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c94d54c-802c-419b-87b5-1650db583a57 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:54.115316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:54.116241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.116621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.116963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] PATH_INFO: `/v3/projects/3b47928a32864affb98164f9081f398c/users/7ee2e074ebf8427c9ca160fac29d9494/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.145463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b47928a32864affb98164f9081f398c, user_id=7ee2e074ebf8427c9ca160fac29d9494, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:54.148752 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.162646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1172/2357] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:54 2022] PUT /identity/v3/projects/3b47928a32864affb98164f9081f398c/users/7ee2e074ebf8427c9ca160fac29d9494/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:54.172971 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.173387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.173724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.202880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] MFA Rules not processed for user `7ee2e074ebf8427c9ca160fac29d9494`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:54.204181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:54.246841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:54.278870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1186/2358] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:54.289450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:54.291197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:54.356914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:54.358339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992215af-16d1-431d-bd50-171ffdd23e76 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:54.359091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:54.362288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:54.362921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.363207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.363485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.372974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:54.374591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.384825 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1173/2359] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:54.432642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615b560-3c19-4cf8-9697-0cb0e0ae1ca2 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:54.437863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:54.438555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:54.438825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:54.439177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] PATH_INFO: `/v3/users/be29d5d3ad50403da073b824eb91adab` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:54.440357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=be29d5d3ad50403da073b824eb91adab)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:54.445432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:54.504641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be29d5d3ad50403da073b824eb91adab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:54.519023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be29d5d3ad50403da073b824eb91adab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:54.519443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be29d5d3ad50403da073b824eb91adab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:54.520672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be29d5d3ad50403da073b824eb91adab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:54.530828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-14071609-cedb-45f2-8aa0-1bb88d98fe4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be29d5d3ad50403da073b824eb91adab'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:54.559929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1187/2360] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:54 2022] DELETE /identity/v3/users/be29d5d3ad50403da073b824eb91adab => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:55.124591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:55.147426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33736157-48e6-4dc6-8e0c-c9f25bde40dd service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:55.196898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:55.197439 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:55.197722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:55.218789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] MFA Rules not processed for user `347e262907da4a83a197aaf42cecf51d`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:55.220023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:55.226086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:55.226729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:55.227009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:55.227329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] PATH_INFO: `/v3/projects/6812aa0ec7234889b3702d8fab9481e6` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:55.228486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6812aa0ec7234889b3702d8fab9481e6)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:55.230661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:55.293255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1188/2361] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:55.294820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6812aa0ec7234889b3702d8fab9481e6', 'request_id': 'req-51153e42-7e27-4c8b-bfa0-903d9a1e7477'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.299565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:55.301324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6812aa0ec7234889b3702d8fab9481e6', 'request_id': 'req-51153e42-7e27-4c8b-bfa0-903d9a1e7477'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.308313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6812aa0ec7234889b3702d8fab9481e6', 'request_id': 'req-51153e42-7e27-4c8b-bfa0-903d9a1e7477'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.325216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] The token cache is being invalidate because project 6812aa0ec7234889b3702d8fab9481e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:55.325731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.326554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.331768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1174/2362] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:55 2022] DELETE /identity/v3/projects/6812aa0ec7234889b3702d8fab9481e6 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:55.332459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:55.334609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:55.351508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:55.356675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a482a9d2-c3f5-4718-8ce6-3e3f5b33c86a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:55.419464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:55.420119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:55.420483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:55.420665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] PATH_INFO: `/v3/users/6f68bda3a203429aae4cf156c7adcadc` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:55.421902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f68bda3a203429aae4cf156c7adcadc)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:55.423897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:55.473866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f68bda3a203429aae4cf156c7adcadc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.482089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f68bda3a203429aae4cf156c7adcadc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.482806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f68bda3a203429aae4cf156c7adcadc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.483438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f68bda3a203429aae4cf156c7adcadc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.491053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-d0398f80-f0cb-48c1-98bb-ccb58f0a87c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f68bda3a203429aae4cf156c7adcadc'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.510319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1189/2363] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:55 2022] DELETE /identity/v3/users/6f68bda3a203429aae4cf156c7adcadc => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:55.694628 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:55.696103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:55.710463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51153e42-7e27-4c8b-bfa0-903d9a1e7477 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:55.773131 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:55.773814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:55.774140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:55.774446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] PATH_INFO: `/v3/projects/7ffa10cfc4ce4d94ab860636aa026789` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:55.775626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ffa10cfc4ce4d94ab860636aa026789)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:55.777604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:55.843885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ffa10cfc4ce4d94ab860636aa026789', 'request_id': 'req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.869709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ffa10cfc4ce4d94ab860636aa026789', 'request_id': 'req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.880918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ffa10cfc4ce4d94ab860636aa026789', 'request_id': 'req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.910872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] The token cache is being invalidate because project 7ffa10cfc4ce4d94ab860636aa026789 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:55.911179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.912775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-9c70023a-9b8e-4e2c-b76a-831e235ecbd3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:55.914467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1175/2364] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:55 2022] DELETE /identity/v3/projects/7ffa10cfc4ce4d94ab860636aa026789 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:55.919852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:55.920284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:55.921027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:55.953969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:55.955498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.043510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1190/2365] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:03:56.053955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.054250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.054525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.081846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:56.083310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.099352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1176/2366] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:56.100283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.118352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09f129-5d22-4140-9c0d-038060c69f49 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.127413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.127604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.128218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.128512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.128789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.129160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.129460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:56.129835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.132824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.149675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1191/2367] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:03:56.151182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c4074-b2e1-4889-8772-6b79d0df0f44 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.152651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.157085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.158031 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.158317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.158585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] PATH_INFO: `/v3/users/0ec732e209d8444ebe8a6a2879d3e39b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.159811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ec732e209d8444ebe8a6a2879d3e39b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.161890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.169083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6278e9-714c-422d-b661-c1e8cff7d9b6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.174811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.176359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.176700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.177090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.178519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.181269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.198007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:56.246777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1192/2368] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/projects => generated 369 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:56.257145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.275290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ec732e209d8444ebe8a6a2879d3e39b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:56.276068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ec732e209d8444ebe8a6a2879d3e39b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:56.292433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ec732e209d8444ebe8a6a2879d3e39b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:56.293617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ec732e209d8444ebe8a6a2879d3e39b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:56.314354 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-aee70630-ca0b-44a5-9b0d-c37d40eedcf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ec732e209d8444ebe8a6a2879d3e39b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:56.322295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=81,keystone:INSERT=3,keystone:DELETE=16 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:56.400595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1177/2369] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:56 2022] DELETE /identity/v3/users/0ec732e209d8444ebe8a6a2879d3e39b => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:56.405081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.405553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.407543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.437200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42643ef7-cd80-4daa-a6b5-14b471639593 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.453061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:56.454338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.550039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.550676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.550958 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.551252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.551874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.554533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.600627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1178/2370] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:56.616086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.617700 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.621402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1193/2371] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/users => generated 285 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:56.625827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.627666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.642965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b698a-3f41-406e-8ded-b15146b59956 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.648640 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.649261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.649545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.650248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.650904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.652635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.664161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1194/2372] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:56.671010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.681600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eea8fe-9c35-4e35-a32e-41a3d80f0d22 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.685873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc0dae-4a3c-42e6-bc06-4f869f849ae0 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.686951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.687580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.687916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.688192 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.688861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.690533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.691490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.692204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.692552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.693860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.704096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1195/2373] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:56.704468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.706021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734b9796-c45c-44d5-b8a5-39e352de88c1 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.708466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.718088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1179/2374] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:56.724146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4176349-42df-4ac3-bb82-96a05119a0ed admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.730448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.731787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.732170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.732999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] PATH_INFO: `/v3/projects/a12a04093dd0461889fa4eaf0bccf0f9/users/58d95e5044444294ae4f2727d46c5e9d/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.750996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a12a04093dd0461889fa4eaf0bccf0f9, user_id=58d95e5044444294ae4f2727d46c5e9d, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.753879 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:56.769893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1196/2375] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:56 2022] PUT /identity/v3/projects/a12a04093dd0461889fa4eaf0bccf0f9/users/58d95e5044444294ae4f2727d46c5e9d/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:56.781610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.781961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.782438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.814493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] MFA Rules not processed for user `58d95e5044444294ae4f2727d46c5e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:56.815823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:56.903518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1180/2376] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:56.907987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:56.958722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:56.982570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd3fe3-4055-4dc7-b054-06a61a39b445 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:56.987546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:56.988165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:56.988449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:56.988728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:56.989369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:56.991919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.028587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1197/2377] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:56 2022] POST /identity/v3/projects => generated 373 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:57.032416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.045025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5387672-3395-49f5-af0a-0a6597808599 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.058172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:57.096817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.098571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.148011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.148771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.149120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.149492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.150293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.152429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.176447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1181/2378] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:57 2022] POST /identity/v3/users => generated 293 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:57.181109 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.193787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788dd286-14b2-4373-a0d8-cfda53a5dbc8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.194779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f6b695-b358-4e7d-ba95-ee66a4c85db6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.199209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.199819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.199964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.200407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.200407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.200580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.200856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.200977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.201131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] PATH_INFO: `/v3/projects/2735a5387cef47149c405e6056ba5b9b` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.202267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2735a5387cef47149c405e6056ba5b9b)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.202629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.204374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.214130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1182/2379] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:57.217642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.228917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fa3dd1-6390-4312-87de-0cdb7cea1c63 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.235023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.235866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.236113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.236386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] PATH_INFO: `/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96/users/2c397a04277d4a3086d7df4c66c97782/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.256119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e55dc4eb8d934ad7b5fef314e0250d96, user_id=2c397a04277d4a3086d7df4c66c97782, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.258804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.264494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2735a5387cef47149c405e6056ba5b9b', 'request_id': 'req-0ae4c011-fa37-4c21-8c04-57a56e79b74e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.271454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2735a5387cef47149c405e6056ba5b9b', 'request_id': 'req-0ae4c011-fa37-4c21-8c04-57a56e79b74e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.277947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2735a5387cef47149c405e6056ba5b9b', 'request_id': 'req-0ae4c011-fa37-4c21-8c04-57a56e79b74e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.287426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1183/2380] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:57 2022] PUT /identity/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96/users/2c397a04277d4a3086d7df4c66c97782/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:57.291242 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.297193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] The token cache is being invalidate because project 2735a5387cef47149c405e6056ba5b9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:57.297489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.298184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.299821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1198/2381] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:57 2022] DELETE /identity/v3/projects/2735a5387cef47149c405e6056ba5b9b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:57.304168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.305483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.353442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d460fabf-8e82-46ef-adf3-322f4da5485d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.358094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.358692 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.358972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.359494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.360182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.361766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.364409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:57.371405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c011-fa37-4c21-8c04-57a56e79b74e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.375945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1184/2382] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:57 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:57.376877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.377159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.377794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.378106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.378406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] PATH_INFO: `/v3/users/2468ceb260864967bc3c6282624a203f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.378729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.379543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2468ceb260864967bc3c6282624a203f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.381647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.391865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cb0262-c2e7-4ced-8163-645e62a88528 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.396814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.397231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.397508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.397780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] PATH_INFO: `/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96/users/2c397a04277d4a3086d7df4c66c97782/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.399710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e55dc4eb8d934ad7b5fef314e0250d96, user_id=2c397a04277d4a3086d7df4c66c97782, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.402358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8b9182-5c87-405e-919c-6ebd0087fb62 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.414575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1185/2383] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:57 2022] PUT /identity/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96/users/2c397a04277d4a3086d7df4c66c97782/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:57.424820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.425245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.425448 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.433588 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2468ceb260864967bc3c6282624a203f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.441008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2468ceb260864967bc3c6282624a203f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.441647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2468ceb260864967bc3c6282624a203f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.442299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2468ceb260864967bc3c6282624a203f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.449550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2468ceb260864967bc3c6282624a203f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:57.454960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] MFA Rules not processed for user `2c397a04277d4a3086d7df4c66c97782`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:57.456160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.468599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1199/2384] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:57 2022] DELETE /identity/v3/users/2468ceb260864967bc3c6282624a203f => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:03:57.502244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:57.540925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1186/2385] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:57 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:57.549619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.551422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.617039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f823b471-2636-498f-9dc7-b479f340274d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.621121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.621825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.622177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.622416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.624288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.694524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.695756 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76430a5-56ca-48dd-ae5b-36b1bf5d0061 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.705387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1200/2386] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:57 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:57.782939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:57.784599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:57.875991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d2aa22-ce32-4507-b603-4e209845733b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:57.936773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:57.937426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:57.937782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:57.938022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] PATH_INFO: `/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:57.939206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5e0f0adbfe3442185a3f1693b70a61f)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:57.941159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:57.994864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5e0f0adbfe3442185a3f1693b70a61f', 'request_id': 'req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.001540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5e0f0adbfe3442185a3f1693b70a61f', 'request_id': 'req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.009394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5e0f0adbfe3442185a3f1693b70a61f', 'request_id': 'req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.025247 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] The token cache is being invalidate because project b5e0f0adbfe3442185a3f1693b70a61f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:58.025644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.027120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-97e26bd0-fc50-4ae7-ad3d-864a2dbf045b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.029081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1187/2387] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:57 2022] DELETE /identity/v3/projects/b5e0f0adbfe3442185a3f1693b70a61f => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:58.035957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.037121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.037411 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.059906 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:58.061381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:58.098683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:58.142469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1201/2388] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:58.151984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.152432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.152569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.174186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:58.175398 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:58.183419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1188/2389] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:58.187618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.197907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caecf377-6adf-4220-8e58-2f0900e23e34 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.202586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.203189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.203511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.203738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.204387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:58.204689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.206305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.218580 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1202/2390] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:58.223254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.232871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e830be6f-5087-41cd-a962-333c70bc12d4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.238147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.238801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.239088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.239363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.240553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.242497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.247244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:58.281009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1189/2391] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/projects => generated 383 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:58.284124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.296009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c12a9-cc4a-4786-be81-9af0c65125a8 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.360713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.361330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.361633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.361909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.362581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.364568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.390021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1203/2392] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:58.391219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.401850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c01f-4f5e-4cc4-9542-80e9f14d3dc7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.406498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.407136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.407417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.407694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.408353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.409954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.423599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1190/2393] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:03:58.424798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.435359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65d90fb-f99a-4977-8805-6915006ef773 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.440528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.440528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.441577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.441801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.442413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.444584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2001fb-39f1-4f1b-a3db-84ba105b5f85 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.458015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1204/2394] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:58.462857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.473094 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0849e1ba-78ea-4254-8188-eeef7fafa3db admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.477831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.478542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.478827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.479104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] PATH_INFO: `/v3/projects/5a8c0e2d33804084896cba3d20b5810c/users/a5f56bc1af8e49e8ab31f58df4eb0862/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.494259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a8c0e2d33804084896cba3d20b5810c, user_id=a5f56bc1af8e49e8ab31f58df4eb0862, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.496914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.510058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1191/2395] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:58 2022] PUT /identity/v3/projects/5a8c0e2d33804084896cba3d20b5810c/users/a5f56bc1af8e49e8ab31f58df4eb0862/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:58.521214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.521499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.521829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.553733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] MFA Rules not processed for user `a5f56bc1af8e49e8ab31f58df4eb0862`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:58.554985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:58.617540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1205/2396] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:58.628652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.630366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:58.688050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d2a71-d9be-450d-a493-8fa8d3b9d7a5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.691720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.692360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.692641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.692915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.704269 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.705694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8169f1a7-3b64-4629-90f5-5a25685a19b6 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.715038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1192/2397] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:03:58.734420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.761221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:58.797060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeebd8d-64ff-430f-821d-d1ab3c0bbff5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:58.801599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:58.802301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.803089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.803361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] PATH_INFO: `/v3/users/a5f56bc1af8e49e8ab31f58df4eb0862` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.804487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5f56bc1af8e49e8ab31f58df4eb0862)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:58.806413 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:58.853269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5f56bc1af8e49e8ab31f58df4eb0862'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.861340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5f56bc1af8e49e8ab31f58df4eb0862'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.862019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5f56bc1af8e49e8ab31f58df4eb0862'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.862685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5f56bc1af8e49e8ab31f58df4eb0862'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.870626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5f56bc1af8e49e8ab31f58df4eb0862'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:58.888184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1206/2398] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:03:58 2022] DELETE /identity/v3/users/a5f56bc1af8e49e8ab31f58df4eb0862 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:58.892551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:58.892891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:58.893173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:58.913642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:58.914861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:58.978711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1193/2399] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:58.989155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:58.990723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:59.048456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38055fdf-80a9-4da4-af1d-6d674d82bc0c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.052698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.053313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.053606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.053857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.063282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.064737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da47645-63d9-4854-928f-067362f5a463 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.074230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1207/2400] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:03:59.339708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.341172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:59.353196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71cb09-185f-4c81-80f8-6491dce4bc99 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.357749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.358422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.358699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.358973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] PATH_INFO: `/v3/projects/5a8c0e2d33804084896cba3d20b5810c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.360350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a8c0e2d33804084896cba3d20b5810c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.362315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.408641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:59.410002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a8c0e2d33804084896cba3d20b5810c', 'request_id': 'req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:59.416809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a8c0e2d33804084896cba3d20b5810c', 'request_id': 'req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:59.423260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a8c0e2d33804084896cba3d20b5810c', 'request_id': 'req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:59.434908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] The token cache is being invalidate because project 5a8c0e2d33804084896cba3d20b5810c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:03:59.435198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:59.435807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e746cf72-2dbe-478b-bd68-4eaf4015e1f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:03:59.437465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1194/2401] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:03:59 2022] DELETE /identity/v3/projects/5a8c0e2d33804084896cba3d20b5810c => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:59.442265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.442733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.442733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.463891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:59.465119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:59.507742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:03:59.527340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1208/2402] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:59.536087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.536813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.537090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.556749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:59.557948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:59.566478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1195/2403] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:59.571079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.580729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5effaa0-ddd7-41a5-bccb-4a10349275ff None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.586019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.586628 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.586909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.587181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.587828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:03:59.588129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.589710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.604838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1209/2404] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:03:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:03:59.605895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.615599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f307f0-19a2-4ad9-934c-c1aa3e35c573 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.620476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.621089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.621367 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.621645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.622493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.624465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.651797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1196/2405] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:03:59 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:03:59.656238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.666785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb5a7d-7cc2-4a09-9bcd-437b0105d714 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.723625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.724884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.725139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.725402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.726056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.727954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.753082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1210/2406] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:03:59 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:03:59.754192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.764662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e47d0-4588-483b-8efd-08b6a76e7ced admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.771933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.772560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.772917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.773182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.773827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.775442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.786070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1197/2407] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:59.790467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.800721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919c60ab-1542-4cb2-85cf-70e251694fd9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.805376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.806036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.806319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.806589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.807226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.808932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf51a7-49ff-49d7-b314-976b04acd0dc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.819552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1211/2408] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:03:59.823573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.833252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9a09d3-a99e-4803-85fa-8d9f9c1a84db admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:03:59.838412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:03:59.839085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.839362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.839635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] PATH_INFO: `/v3/projects/9205499b83d14ee2b0baba543b3e19b4/users/3a7bbabb9e0f46eca84c8e83022431de/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.855037 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9205499b83d14ee2b0baba543b3e19b4, user_id=3a7bbabb9e0f46eca84c8e83022431de, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:03:59.857691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:03:59.870652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1198/2409] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:03:59 2022] PUT /identity/v3/projects/9205499b83d14ee2b0baba543b3e19b4/users/3a7bbabb9e0f46eca84c8e83022431de/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:03:59.880609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:03:59.880892 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:03:59.881158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:03:59.908138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] MFA Rules not processed for user `3a7bbabb9e0f46eca84c8e83022431de`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:03:59.909323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:03:59.972365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1212/2410] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:03:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:03:59.983990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:03:59.985926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:00.051218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca5db4-6f38-4624-9fd7-c9b50cbc45f6 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:00.054997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:00.055612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:00.055895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:00.056164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:00.064751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:00.066251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52de3ecb-9c76-434c-b257-a5e7019ca8ce service glance] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:00.076044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1199/2411] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:03:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:03.482751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:03.484648 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:03.533146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:03.570540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f2d79d-d7cd-41a6-9cb8-8d1f6a671146 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:03.576106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:03.576873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:03.577793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:03.578841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] PATH_INFO: `/v3/users/58d95e5044444294ae4f2727d46c5e9d` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:03.582036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=58d95e5044444294ae4f2727d46c5e9d)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:03.586961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:03.650900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58d95e5044444294ae4f2727d46c5e9d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:03.664766 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58d95e5044444294ae4f2727d46c5e9d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:03.665475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58d95e5044444294ae4f2727d46c5e9d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:03.666449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58d95e5044444294ae4f2727d46c5e9d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:03.674682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58d95e5044444294ae4f2727d46c5e9d'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:03.700321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1213/2412] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:03 2022] DELETE /identity/v3/users/58d95e5044444294ae4f2727d46c5e9d => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:03.704977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:03.705521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:03.705741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:03.727048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:03.728468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:03.818478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1200/2413] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:03.830253 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:03.831642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:03.897714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8cda55-6d51-470d-8328-293aceedcb2e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:03.901403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:03.902057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:03.902339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:03.902609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:03.911927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:03.913364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:03.922722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1214/2414] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:04.178191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:04.180298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:04.194079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b25a68-9a66-4046-8257-6b2a39cfa0fe None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:04.195839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:04.197472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:04.199376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:04.203543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:04.203543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:04.203543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] PATH_INFO: `/v3/users/7ee2e074ebf8427c9ca160fac29d9494` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:04.203543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ee2e074ebf8427c9ca160fac29d9494)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:04.204002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:04.215970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:04.219961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aecb320-9dc6-498b-8468-622ef76f94dc service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:04.227619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:04.228275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:04.228556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:04.228830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] PATH_INFO: `/v3/projects/a12a04093dd0461889fa4eaf0bccf0f9` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:04.230090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a12a04093dd0461889fa4eaf0bccf0f9)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:04.232037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:04.265061 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ee2e074ebf8427c9ca160fac29d9494'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.265710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ee2e074ebf8427c9ca160fac29d9494'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.275685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ee2e074ebf8427c9ca160fac29d9494'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.276370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ee2e074ebf8427c9ca160fac29d9494'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.283574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a8227c26-4aa8-4c5d-b9a1-70b726f164fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ee2e074ebf8427c9ca160fac29d9494'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.291159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a12a04093dd0461889fa4eaf0bccf0f9', 'request_id': 'req-ea27c916-6a2b-4fe4-b263-3e9342b2a373'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.297260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a12a04093dd0461889fa4eaf0bccf0f9', 'request_id': 'req-ea27c916-6a2b-4fe4-b263-3e9342b2a373'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.298793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:04.303512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1201/2415] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:04 2022] DELETE /identity/v3/users/7ee2e074ebf8427c9ca160fac29d9494 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:04.307549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:04.307844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:04.308127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:04.308790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a12a04093dd0461889fa4eaf0bccf0f9', 'request_id': 'req-ea27c916-6a2b-4fe4-b263-3e9342b2a373'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.324649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] The token cache is being invalidate because project a12a04093dd0461889fa4eaf0bccf0f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:04.324649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.325124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.327561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1215/2416] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:04 2022] DELETE /identity/v3/projects/a12a04093dd0461889fa4eaf0bccf0f9 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:04.334085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:04.334312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:04.334420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:04.334519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:04.429116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1202/2417] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:04.444891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea27c916-6a2b-4fe4-b263-3e9342b2a373 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:04.450328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:04.455882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:04.463587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:04.465570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:04.465862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:04.466164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] PATH_INFO: `/v3/users/2c397a04277d4a3086d7df4c66c97782` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:04.469414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c397a04277d4a3086d7df4c66c97782)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:04.471371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:04.540314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ebb9ca-e98c-4d69-8705-d70c62ad8db1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:04.544027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:04.544759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:04.545207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:04.545515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:04.555688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c397a04277d4a3086d7df4c66c97782'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.557338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:04.559607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:04.571964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c397a04277d4a3086d7df4c66c97782'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.572690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c397a04277d4a3086d7df4c66c97782'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.573384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c397a04277d4a3086d7df4c66c97782'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.574428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1203/2418] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:04.585776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c397a04277d4a3086d7df4c66c97782'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:04.612493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1216/2419] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:04 2022] DELETE /identity/v3/users/2c397a04277d4a3086d7df4c66c97782 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:05.002293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.004246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.017860 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec228e-9b64-4134-aadb-495adb7ffb96 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.068066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:05.078462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.080078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.116243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.117223 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.117561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.118085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] PATH_INFO: `/v3/projects/3b47928a32864affb98164f9081f398c` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.119263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b47928a32864affb98164f9081f398c)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.121270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.171415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c961ed-cd5a-4685-aa4c-95a6f42942a0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.175211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b47928a32864affb98164f9081f398c', 'request_id': 'req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.181011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b47928a32864affb98164f9081f398c', 'request_id': 'req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.181777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:05.189077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b47928a32864affb98164f9081f398c', 'request_id': 'req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.205257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] The token cache is being invalidate because project 3b47928a32864affb98164f9081f398c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:05.205592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.206412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3dcd25e0-8aa0-4ab0-bd49-962ce5696c58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.208846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1204/2420] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:05 2022] DELETE /identity/v3/projects/3b47928a32864affb98164f9081f398c => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:05.213652 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc65e00-b636-40d7-892e-ada5cc91ade7 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.213941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc65e00-b636-40d7-892e-ada5cc91ade7 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.214268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc65e00-b636-40d7-892e-ada5cc91ade7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.259133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.259874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.260153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.260423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] PATH_INFO: `/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.261563 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e55dc4eb8d934ad7b5fef314e0250d96)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.263542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.269796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-6cc65e00-b636-40d7-892e-ada5cc91ade7 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:05.271657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6cc65e00-b636-40d7-892e-ada5cc91ade7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.282824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1205/2421] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:05.291911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.292286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.292637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.321841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:05.323113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.325196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e55dc4eb8d934ad7b5fef314e0250d96', 'request_id': 'req-b4247e09-656f-4b03-be0d-92ad8434c54b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.332042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e55dc4eb8d934ad7b5fef314e0250d96', 'request_id': 'req-b4247e09-656f-4b03-be0d-92ad8434c54b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.338927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e55dc4eb8d934ad7b5fef314e0250d96', 'request_id': 'req-b4247e09-656f-4b03-be0d-92ad8434c54b'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.353148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] The token cache is being invalidate because project e55dc4eb8d934ad7b5fef314e0250d96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:05.353364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.354181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b4247e09-656f-4b03-be0d-92ad8434c54b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:05.355898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1217/2422] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:05 2022] DELETE /identity/v3/projects/e55dc4eb8d934ad7b5fef314e0250d96 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:05.360568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978eff8-147e-4ad8-9000-f8a82ff96efa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.360857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978eff8-147e-4ad8-9000-f8a82ff96efa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.361137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7978eff8-147e-4ad8-9000-f8a82ff96efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.384132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7978eff8-147e-4ad8-9000-f8a82ff96efa None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:05.385418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7978eff8-147e-4ad8-9000-f8a82ff96efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.399268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1206/2423] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:05.403529 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.414499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985d3915-0a32-4320-8cb0-d423b9b574ca None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.420268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.420894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.421175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.421450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.422280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:05.422585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.424197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.439349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1207/2424] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:04:05.444271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.454592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19854f9-f5d2-4b59-85d0-4c9db7ab46fd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.457136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1218/2425] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:05.461683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.462328 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.462608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.462879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.463508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.465544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.466614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.466908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.467191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.491153 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:05.492362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:05.497297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1208/2426] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:05.501191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1219/2427] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:05.504400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.508147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.518955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029e2f0-f1b0-43c4-8b30-b4f44655933d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.529384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:05.568864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:05.586088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebfcc16-6233-49c7-a0a2-3c0364e32446 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.593824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.594497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.594778 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.595082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.596157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.598644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.606982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.607663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.607948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.608381 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.609052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:05.610121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.611753 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.627634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1220/2428] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:04:05.632190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1209/2429] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/users => generated 293 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:05.632367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.633074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.643842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70e309-a75e-4d90-96f5-d206beeaaf22 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.646087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fe71f3-2c7f-472a-a1fd-9bcc12d827d5 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.649251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.650180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.650461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.650902 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.651544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.653843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.655898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.656749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.657193 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.657630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.658481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.661078 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.680229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1210/2430] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:05.681346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.688920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1221/2431] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:05.693297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.699363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfc08e4-f860-49e1-bfe8-1a0e482200d4 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.770187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.771376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.771619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.771888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.772590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.774567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.789784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1211/2432] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:05.790846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.802150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b9a17-e2cb-4bbb-bd18-9eb48ff0e349 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.808220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.808892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.809173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.809443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] PATH_INFO: `/v3/projects/19246f2f2f1844ec8dd054a65369c26f/users/950a5cea22194177b39c5d61ae1a3f17/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.817603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78475384-ba61-4947-aca7-ce33f49e69a4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.829245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:05.830009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.830405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.830772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.831502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.834378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.841615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19246f2f2f1844ec8dd054a65369c26f, user_id=950a5cea22194177b39c5d61ae1a3f17, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:05.845023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35e6245-fef6-4e7f-905d-88f155a9aed5 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:05.867343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1212/2433] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:05 2022] PUT /identity/v3/projects/19246f2f2f1844ec8dd054a65369c26f/users/950a5cea22194177b39c5d61ae1a3f17/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:05.877409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1222/2434] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/users => generated 288 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:05.879481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:05.879771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:05.881333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:05.882522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:05.893932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195b4d21-8e38-4c32-9a64-86b6a3f32ba3 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:05.963827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] MFA Rules not processed for user `950a5cea22194177b39c5d61ae1a3f17`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:05.967364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:06.010061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.010703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.010985 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.011264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.012186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.015085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.042206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1223/2435] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:06.049273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.063272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.063853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc0115-5e2e-4faa-8744-f3fb7770bb36 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.072485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1213/2436] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:06.073450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.082305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.082948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.083231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.085293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.085995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.087632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.089112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dc5365-465c-45a8-af2e-c18b6f850c7b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.094554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.095173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.095470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.095736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.096376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.098335 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.099682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1224/2437] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:06.115333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.128296 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a29bd-8307-43c7-8aae-903362f73523 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.134266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.134986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.135277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.135555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] PATH_INFO: `/v3/projects/2bf16c47a64147ddab873f2181b4daad/users/f819f31093ac49de8d4e226ecd35f60f/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.150846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1214/2438] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/projects => generated 385 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:06.153458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.155898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bf16c47a64147ddab873f2181b4daad, user_id=f819f31093ac49de8d4e226ecd35f60f, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.158721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6629c4-b7ec-4094-aed7-f3f5b89fbe03 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.172351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.178634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1225/2439] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:06 2022] PUT /identity/v3/projects/2bf16c47a64147ddab873f2181b4daad/users/f819f31093ac49de8d4e226ecd35f60f/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:06.207795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.209682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.210176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.242233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b317b3e-512b-4600-917f-ea5d780bb324 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.242814 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] MFA Rules not processed for user `f819f31093ac49de8d4e226ecd35f60f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:06.244030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:06.306254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.310915 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.311456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.311730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.312400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.314380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.323288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1226/2440] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:06.327926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.339243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1215/2441] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/users => generated 293 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:06.342851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.343014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f92f6-8791-4af7-b5d2-a518e4e6cc03 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.347783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.348390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.348703 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.349009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.349650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.351714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.353230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab173f84-876a-41d1-9a39-531927935588 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.358513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.359157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.359701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.359968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.360610 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.362274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.375553 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1216/2442] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:06.376462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.379490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1227/2443] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:06.384057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.386886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75098d63-53fe-44fd-b58b-5a08c7372350 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.449048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ba09a1-e082-4059-8298-fbc6b2f714a3 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.452363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.453021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.453302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.453576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.454263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.455907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.460599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.461234 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.461541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.461811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.462262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.462486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.464844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.467894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1217/2444] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:06.471967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.482582 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361b0fb-1722-47f7-9e13-20f0fefc2a6a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.487719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.488396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.488679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.488974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] PATH_INFO: `/v3/projects/f82b9e0825574616a481763fe3ea6c82/users/6614de0f3e5849feb52e3678df1e3205/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.496371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1228/2445] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/users => generated 289 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:06.497433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.507908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e9810c-93c3-4ac6-921f-5f4e9b4018b9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.508045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=f82b9e0825574616a481763fe3ea6c82, user_id=6614de0f3e5849feb52e3678df1e3205, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.510707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.512601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.513199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.513484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.513765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.514447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.516031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76737296-8a0a-4b45-bf82-a0e2d30ca0bf admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.523994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1218/2446] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:06 2022] PUT /identity/v3/projects/f82b9e0825574616a481763fe3ea6c82/users/6614de0f3e5849feb52e3678df1e3205/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:06.533010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1229/2447] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:06.533142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.535794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.536093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.536456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.569085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] MFA Rules not processed for user `6614de0f3e5849feb52e3678df1e3205`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:06.570410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:06.606008 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc79f04-7500-47fe-a435-8090708252ba admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.607143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.611221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.611830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.612108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.612383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.613038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.615342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.626006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1219/2448] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:06.629758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.640302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cb6652-f9b3-4d16-bc41-0c73b5119b19 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.644995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.645649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.645997 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.646250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] PATH_INFO: `/v3/projects/dfde7d744c2b423eac77b7aff6662acf/users/896764cc47544a93ab04cc6b90490fc1/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.649007 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1230/2449] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:06.653637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.661727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfde7d744c2b423eac77b7aff6662acf, user_id=896764cc47544a93ab04cc6b90490fc1, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.663807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cbd50e-9d73-4ccb-afbb-7100bd73910d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.664539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49d234d-9c3c-41d5-82d9-2b30c013c51e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.668837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.669529 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.669755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.670055 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.670700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.672865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.683191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1220/2450] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:06 2022] PUT /identity/v3/projects/dfde7d744c2b423eac77b7aff6662acf/users/896764cc47544a93ab04cc6b90490fc1/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:06.693764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.694087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.694375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.708194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1231/2451] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:06.709224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.724793 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] MFA Rules not processed for user `896764cc47544a93ab04cc6b90490fc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:06.726087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:06.774757 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f0351-b392-4de3-8287-f7b3ec76e5cf admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.779521 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.780135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.780416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.780696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.781326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.783348 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.804213 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1221/2452] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:06.807223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1232/2453] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:06.809194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.810837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.820378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06e869c-c554-49ae-9061-5c2da2da3947 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.821177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729883a-f119-48df-9444-43898d4c2533 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.825042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.825660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.825943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.826263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.826518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.826896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.827126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.827416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.827679 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.828331 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.828942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.830158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.840873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1233/2454] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:06.845307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.855398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8e971-7e3b-418d-bcea-3f1312c5e477 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.860608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1222/2455] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:06.862312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.880703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.918603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.919404 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.919822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.920127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] PATH_INFO: `/v3/projects/50dae096beb04ffab40e00d635afe916/users/98cec4a89fda40d1914a31faa7574fc2/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.930896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f4445-6ba6-487d-a714-391894fdd1ad admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:06.936388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] RBAC: Authorizing `identity:create_grant(project_id=50dae096beb04ffab40e00d635afe916, user_id=98cec4a89fda40d1914a31faa7574fc2, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.936761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:06.937260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:06.937527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:06.937944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:06.938619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:06.939535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.941737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:06.954089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1234/2456] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:06 2022] PUT /identity/v3/projects/50dae096beb04ffab40e00d635afe916/users/98cec4a89fda40d1914a31faa7574fc2/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:06.957801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.969192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1223/2457] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:06 2022] POST /identity/v3/users => generated 295 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:06.970240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:06.980557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:06.982916 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ade849d-e8e5-4288-bae6-f26bc27f5403 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.049132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279b3ac-eda3-48da-bad6-5123fc06031b admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.055802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.056058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.056719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.056842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.057135 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.057264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.057415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.057612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.058314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.058639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.060849 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.061888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.075121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1235/2458] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:07.075331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1224/2459] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:07.077535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.077697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.089723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bcabac-1cb1-47b5-9518-79daf8d58ca3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.090125 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5efa89f-b5c5-4843-8efb-e04fd1d483b6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.093912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.093912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.093912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.094519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] PATH_INFO: `/v3/projects/17e8edbb491942b5bca48b6ed8f78e95/users/5444607f7e5e4de58304185f33f26ab5/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.095491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.096178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.096460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.096728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] PATH_INFO: `/v3/projects/50dae096beb04ffab40e00d635afe916/users/98cec4a89fda40d1914a31faa7574fc2/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.098782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50dae096beb04ffab40e00d635afe916, user_id=98cec4a89fda40d1914a31faa7574fc2, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.101991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf010c3-1b25-4f62-84ef-6dfb668841a0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.114528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1236/2460] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:07 2022] PUT /identity/v3/projects/50dae096beb04ffab40e00d635afe916/users/98cec4a89fda40d1914a31faa7574fc2/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:07.115285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17e8edbb491942b5bca48b6ed8f78e95, user_id=5444607f7e5e4de58304185f33f26ab5, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.118736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.125180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.125501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.125780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.139499 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1225/2461] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:07 2022] PUT /identity/v3/projects/17e8edbb491942b5bca48b6ed8f78e95/users/5444607f7e5e4de58304185f33f26ab5/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:07.143483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.158659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] MFA Rules not processed for user `98cec4a89fda40d1914a31faa7574fc2`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:07.159915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:07.230904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802108ed-cfaa-4681-b56c-30ac29a753a9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.238814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.239856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.240244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.240699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.241538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.253968 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.254189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1237/2462] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:07 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:07.261034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1226/2463] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:07 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:07.263195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.265298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:07.270312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.288230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57aa6f87-fee2-4049-ae9a-654914fcf1d3 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.293430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.294149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.294426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.294704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] PATH_INFO: `/v3/projects/17e8edbb491942b5bca48b6ed8f78e95/users/5444607f7e5e4de58304185f33f26ab5/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.296661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] RBAC: Authorizing `identity:create_grant(project_id=17e8edbb491942b5bca48b6ed8f78e95, user_id=5444607f7e5e4de58304185f33f26ab5, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.299548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5e583b-60c3-48b3-b64d-6e675cdc236c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.306506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:07.316140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1227/2464] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:07 2022] PUT /identity/v3/projects/17e8edbb491942b5bca48b6ed8f78e95/users/5444607f7e5e4de58304185f33f26ab5/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:07.353828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1827bde-5596-41bb-b6ce-f11e8663ce3c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.390803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.391446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.391932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.408899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:07.426939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.427716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.428099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.428443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.442461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] MFA Rules not processed for user `5444607f7e5e4de58304185f33f26ab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:07.443681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:07.499247 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.500829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.512711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1238/2465] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:07.525938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1228/2466] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:07 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:07.538207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:07.539920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:07.553949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4aab7-edc4-4a1b-9be2-fd5c044f0429 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:07.557862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:07.558581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:07.558939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:07.559280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:07.622807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:07.624258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:07.634483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1239/2467] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:10.118795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:10.131789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0dde46-e9ce-45e4-9ee6-8a97de120868 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:10.135746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:10.136396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:10.136680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:10.136953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:10.212113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:10.213561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:10.228697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1229/2468] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:10.277104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:10.285808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a22f11-abe2-452f-a17f-af6c65df9ae5 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:10.289552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:10.290185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:10.290497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:10.290771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:10.355824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:10.357569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:10.367975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1240/2469] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:10 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:10.792973 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:10.802877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89815d45-4f01-4995-96f4-c002c1d03242 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:10.807110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:10.807722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:10.808009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:10.808276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:10.972601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:10.974741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15930ec4-7675-44b5-9e72-8aab9d4f288b service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:10.999386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1230/2470] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:10 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:11.069099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:11.080134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682376-0a1e-4c63-aacd-1eb706f3997f service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:11.089526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:11.098118 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:11.099790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:11.100305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:11.119751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:11.121511 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:11.133211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1241/2471] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:11 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:14.602221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:14.602665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:14.602922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:14.623632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:14.624831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:14.638178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:14.684403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1231/2472] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:04:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:14.727063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:14.728505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:14.749443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:14.797706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c23236-b328-41fc-899d-a7f3c3babdec service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:14.803273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:14.803888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:14.804169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:14.804438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:14.812752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:14.814959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e08cefa-00ff-4011-b361-ae4a7d1f04a5 service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:14.824650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1242/2473] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:04:15.131805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:15.237084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ffc167-4174-4f96-8499-fc145ba9306d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:15.243355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:15.244200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.244476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.244745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] PATH_INFO: `/v3/users/950a5cea22194177b39c5d61ae1a3f17` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.245913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=950a5cea22194177b39c5d61ae1a3f17)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:15.255672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:15.316158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '950a5cea22194177b39c5d61ae1a3f17'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.316821 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '950a5cea22194177b39c5d61ae1a3f17'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.329195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '950a5cea22194177b39c5d61ae1a3f17'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.330045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '950a5cea22194177b39c5d61ae1a3f17'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.338788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '950a5cea22194177b39c5d61ae1a3f17'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.360344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1232/2474] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:15 2022] DELETE /identity/v3/users/950a5cea22194177b39c5d61ae1a3f17 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:15.364746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7b6eb-b6e3-4a2d-ae0c-061cdc4e46e6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.365119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7b6eb-b6e3-4a2d-ae0c-061cdc4e46e6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.365681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc7b6eb-b6e3-4a2d-ae0c-061cdc4e46e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.393258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6bc7b6eb-b6e3-4a2d-ae0c-061cdc4e46e6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:15.394598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6bc7b6eb-b6e3-4a2d-ae0c-061cdc4e46e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:15.471188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1243/2475] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:15.477962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:15.479635 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:15.535966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.536279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.536553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.539726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:15.554147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12432fba-74ff-4c3a-98f6-8a879f615d7c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:15.557425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:15.557808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:15.558496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.558652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:15.558780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.559056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.568980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:15.570498 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:15.580153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1233/2476] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:15.613676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1244/2477] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:04:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:15.627760 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:15.629502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:15.721537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f63ff-43ba-4916-9f70-39193d3a8943 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:15.727638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:15.728746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.728746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.729002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.745388 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:15.747019 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:15.755917 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1234/2478] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:04:15.872036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:15.873761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:15.891681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef8d5c8-d914-4f99-8e2a-c0b139f8e31c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:15.896635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:15.897291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:15.897576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:15.897854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] PATH_INFO: `/v3/projects/19246f2f2f1844ec8dd054a65369c26f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:15.899139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] RBAC: Authorizing `identity:delete_project(project_id=19246f2f2f1844ec8dd054a65369c26f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:15.901081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:15.951887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:15.959606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19246f2f2f1844ec8dd054a65369c26f', 'request_id': 'req-12c85435-a5e0-40c2-a3ac-ba550013295e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.977238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19246f2f2f1844ec8dd054a65369c26f', 'request_id': 'req-12c85435-a5e0-40c2-a3ac-ba550013295e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:15.988769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19246f2f2f1844ec8dd054a65369c26f', 'request_id': 'req-12c85435-a5e0-40c2-a3ac-ba550013295e'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.005432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] The token cache is being invalidate because project 19246f2f2f1844ec8dd054a65369c26f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:16.005914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.006562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-12c85435-a5e0-40c2-a3ac-ba550013295e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.008137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1245/2479] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:15 2022] DELETE /identity/v3/projects/19246f2f2f1844ec8dd054a65369c26f => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:16.012834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.028844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee503b91-eb86-428a-8245-90566452c233 service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.055702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.056034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.056294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.079875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:16.081229 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.119288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.120403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.124134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.124134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] PATH_INFO: `/v3/users/6614de0f3e5849feb52e3678df1e3205` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.124134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6614de0f3e5849feb52e3678df1e3205)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.125587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.198770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6614de0f3e5849feb52e3678df1e3205'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.199526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6614de0f3e5849feb52e3678df1e3205'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.210364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6614de0f3e5849feb52e3678df1e3205'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.211374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6614de0f3e5849feb52e3678df1e3205'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.221374 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6614de0f3e5849feb52e3678df1e3205'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.237532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:16.248111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1235/2480] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:16 2022] DELETE /identity/v3/users/6614de0f3e5849feb52e3678df1e3205 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:16.283542 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1246/2481] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:04:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:16.296149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.297670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.372805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.374554 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.384338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d0dc74-1797-44c9-8e2c-8a9e5bada279 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.389772 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.390226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55086d0-8af8-4f16-95b5-3f458b739668 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.390462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.390744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.391018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.461868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.463791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.474319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1236/2482] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 11:04:16.477176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.478107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.478446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.478646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] PATH_INFO: `/v3/users/f819f31093ac49de8d4e226ecd35f60f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.479830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f819f31093ac49de8d4e226ecd35f60f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.481874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.555802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f819f31093ac49de8d4e226ecd35f60f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.569961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.570577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f819f31093ac49de8d4e226ecd35f60f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.571311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f819f31093ac49de8d4e226ecd35f60f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.571596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.572176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f819f31093ac49de8d4e226ecd35f60f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.588036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-36c5778d-6dce-440d-9f43-0b6eeb8e4624 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f819f31093ac49de8d4e226ecd35f60f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.594068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f94552a-a637-4796-bbe9-87262d5bde3c service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.601070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.601829 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.602225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.602723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] PATH_INFO: `/v3/projects/f82b9e0825574616a481763fe3ea6c82` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.603843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] RBAC: Authorizing `identity:delete_project(project_id=f82b9e0825574616a481763fe3ea6c82)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.606316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.622267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:16.632139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1247/2483] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:16 2022] DELETE /identity/v3/users/f819f31093ac49de8d4e226ecd35f60f => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:16.633576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.633965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.634356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.659001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:16.660222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.691174 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f82b9e0825574616a481763fe3ea6c82', 'request_id': 'req-12d0c84b-e075-43f1-a0d5-ab75995cfabb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.699639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f82b9e0825574616a481763fe3ea6c82', 'request_id': 'req-12d0c84b-e075-43f1-a0d5-ab75995cfabb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.707057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f82b9e0825574616a481763fe3ea6c82', 'request_id': 'req-12d0c84b-e075-43f1-a0d5-ab75995cfabb'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.739699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] The token cache is being invalidate because project f82b9e0825574616a481763fe3ea6c82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:16.740001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.749938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1248/2484] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:16.753745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.758786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1237/2485] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:16 2022] DELETE /identity/v3/projects/f82b9e0825574616a481763fe3ea6c82 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:16.759871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.761546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.766805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:16.768313 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.788070 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa975b3-c3f8-4e9c-8e02-44d90566ab70 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.795953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.796962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.797447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.798339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] PATH_INFO: `/v3/users/98cec4a89fda40d1914a31faa7574fc2` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.799952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98cec4a89fda40d1914a31faa7574fc2)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.803057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.826348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:16.870162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98cec4a89fda40d1914a31faa7574fc2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.876068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d0c84b-e075-43f1-a0d5-ab75995cfabb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:16.881236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:16.881933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.882310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.882646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.884907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:16.895368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98cec4a89fda40d1914a31faa7574fc2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.898371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98cec4a89fda40d1914a31faa7574fc2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.899340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98cec4a89fda40d1914a31faa7574fc2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.899711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:16.901218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bb0cb0-adfb-4d97-a1f1-c920158c8d82 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:16.908271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98cec4a89fda40d1914a31faa7574fc2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:16.912357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1238/2486] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:04:16.917805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:16.918391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:16.918736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:16.927445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1249/2487] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:16 2022] DELETE /identity/v3/users/98cec4a89fda40d1914a31faa7574fc2 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:16.939749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:16.941040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.013525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1239/2488] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:04:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:17.029479 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.032073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.135553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407ae93c-2ead-4781-959f-fcec9a72ea37 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:17.140770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:17.141386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.141671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.141946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.154002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:17.155612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db431684-4661-41e9-ba00-e5bb72af7703 service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:17.173513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1250/2489] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:04:17.233025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.235469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.350735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed98a6-3201-4d27-8802-31ad76607406 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:17.362183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:17.363266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.363383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.363775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] PATH_INFO: `/v3/projects/50dae096beb04ffab40e00d635afe916` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.366536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50dae096beb04ffab40e00d635afe916)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:17.368535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:17.458904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50dae096beb04ffab40e00d635afe916', 'request_id': 'req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.464942 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50dae096beb04ffab40e00d635afe916', 'request_id': 'req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.472249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50dae096beb04ffab40e00d635afe916', 'request_id': 'req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.481433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:17.509893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] The token cache is being invalidate because project 50dae096beb04ffab40e00d635afe916 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:17.510244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.511544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.514077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1240/2490] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:17 2022] DELETE /identity/v3/projects/50dae096beb04ffab40e00d635afe916 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:17.523787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67640b5f-f6a4-4bf7-86bd-30e8de5cb990 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.524465 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67640b5f-f6a4-4bf7-86bd-30e8de5cb990 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.525018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67640b5f-f6a4-4bf7-86bd-30e8de5cb990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.535975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:17.562607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-67640b5f-f6a4-4bf7-86bd-30e8de5cb990 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:17.564078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-67640b5f-f6a4-4bf7-86bd-30e8de5cb990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.624897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.626334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.690281 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1251/2491] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:17.698139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.698424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.698712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.721545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:17.722781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.732134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1252/2492] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:17.737033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.748021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470539a-94d6-4399-8f8b-1cd26e148ef6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:17.753875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:17.754605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.754905 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.755176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.755822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:17.756723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:17.758677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:17.761027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc1e73b-6e84-47bd-8cfc-1783a3ec6624 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:17.768830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:17.769411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.769794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.770289 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] PATH_INFO: `/v3/projects/2bf16c47a64147ddab873f2181b4daad` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.772376 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bf16c47a64147ddab873f2181b4daad)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:17.774495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:17.780240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1253/2493] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:04:17.781366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.793547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e723f36-c8c3-4346-9670-ea29e426f997 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:17.798485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:17.799119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:17.799396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:17.799664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:17.800524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:17.802876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:17.843619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1254/2494] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:17 2022] POST /identity/v3/projects => generated 363 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:17.847794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.886919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bf16c47a64147ddab873f2181b4daad', 'request_id': 'req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.899512 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bf16c47a64147ddab873f2181b4daad', 'request_id': 'req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.906921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bf16c47a64147ddab873f2181b4daad', 'request_id': 'req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.934454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] The token cache is being invalidate because project 2bf16c47a64147ddab873f2181b4daad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:17.934761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.936479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:17.937270 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1241/2495] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:17 2022] DELETE /identity/v3/projects/2bf16c47a64147ddab873f2181b4daad => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:17.940962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:17.942524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:17.969175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd02da3b-07af-4900-bb98-a529d49881f4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.024156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ead848-28ef-4a03-a93a-2fb3f3d0dd00 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.031201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.031786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.032154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.032352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] PATH_INFO: `/v3/users/896764cc47544a93ab04cc6b90490fc1` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.033945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] RBAC: Authorizing `identity:delete_user(user_id=896764cc47544a93ab04cc6b90490fc1)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.036887 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.071433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.071782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.072146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.075824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.076159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.078263 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.079831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:18.113531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '896764cc47544a93ab04cc6b90490fc1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:18.114812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '896764cc47544a93ab04cc6b90490fc1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:18.116313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:18.125033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1255/2496] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:17 2022] POST /identity/v3/users => generated 282 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:18.131262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:18.133071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:18.133835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '896764cc47544a93ab04cc6b90490fc1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:18.135613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '896764cc47544a93ab04cc6b90490fc1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:18.151631 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6f504-e776-4722-a1ee-41fcc354ded4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.154021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '896764cc47544a93ab04cc6b90490fc1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:18.163726 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.164236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.164514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.164780 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.165508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.169066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.188793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1256/2497] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:18.189709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:18.191723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:18.192414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1242/2498] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:17 2022] DELETE /identity/v3/users/896764cc47544a93ab04cc6b90490fc1 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:18.208132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374b807-718a-4725-bdb0-11649b0af6e1 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.317882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.318671 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.318952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.319226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.320613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.322399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e991e1-a080-46a8-bba1-3df60ae0194b admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.347047 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1257/2499] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:18.351959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:18.378895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527e749-45b0-453d-9a6b-61e505c8f91a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.385117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.385958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.386297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.386630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] PATH_INFO: `/v3/projects/36a584cd24cd42179665ad2d6b3eea46/users/777461af3f1e4c218ac8278a132698bd/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.416622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36a584cd24cd42179665ad2d6b3eea46, user_id=777461af3f1e4c218ac8278a132698bd, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.420939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.441415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1243/2500] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:18 2022] PUT /identity/v3/projects/36a584cd24cd42179665ad2d6b3eea46/users/777461af3f1e4c218ac8278a132698bd/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:18.453230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.453738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.463262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.513881 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] MFA Rules not processed for user `777461af3f1e4c218ac8278a132698bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:18.515189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:18.607159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1258/2501] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:18 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:18.615046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:18.616928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:18.717882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7be4d2-16cb-454c-b5fb-bb604cdd4223 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:18.722802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:18.724914 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:18.726228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:18.726620 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:18.737381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:18.738979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:18.750442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1244/2502] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:18 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:18.870643 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:18.874528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:19.057559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d601bd4e-122f-4d01-b590-1233fb9c83e0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:19.064346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:19.065002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.066045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.066344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] PATH_INFO: `/v3/projects/dfde7d744c2b423eac77b7aff6662acf` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:19.067524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfde7d744c2b423eac77b7aff6662acf)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:19.073260 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:19.101938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:19.153878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfde7d744c2b423eac77b7aff6662acf', 'request_id': 'req-f3fc1f40-e165-4002-85de-ee7f0df101c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.166398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfde7d744c2b423eac77b7aff6662acf', 'request_id': 'req-f3fc1f40-e165-4002-85de-ee7f0df101c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.199475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfde7d744c2b423eac77b7aff6662acf', 'request_id': 'req-f3fc1f40-e165-4002-85de-ee7f0df101c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.221931 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:19.224255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:19.227392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] The token cache is being invalidate because project dfde7d744c2b423eac77b7aff6662acf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:19.227744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.228440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.230206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1259/2503] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:18 2022] DELETE /identity/v3/projects/dfde7d744c2b423eac77b7aff6662acf => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:19.234198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:19.237596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:19.320784 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc1f40-e165-4002-85de-ee7f0df101c4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:19.326442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:19.327233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.327515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.327790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] PATH_INFO: `/v3/users/5444607f7e5e4de58304185f33f26ab5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:19.329337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5444607f7e5e4de58304185f33f26ab5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:19.331890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:19.337324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98265acf-9de3-4325-bc53-978f153965b5 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:19.389248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:19.405250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:19.406543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5444607f7e5e4de58304185f33f26ab5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.406999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.407479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.408287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:19.411803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:19.419154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5444607f7e5e4de58304185f33f26ab5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.420900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5444607f7e5e4de58304185f33f26ab5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.422406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5444607f7e5e4de58304185f33f26ab5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.431593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5444607f7e5e4de58304185f33f26ab5'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.452205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1260/2504] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:19 2022] DELETE /identity/v3/users/5444607f7e5e4de58304185f33f26ab5 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:19.495796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:19.497332 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:19.509872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1245/2505] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:19 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:19.747285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:19.749005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:19.768957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:19.781055 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fdd61-a462-4b2e-9cd9-e4c0e326d214 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:19.784560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:19.847330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f98bc81-779c-453d-b668-e18c98494074 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:19.855360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:19.856103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.856449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.857191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] PATH_INFO: `/v3/users/e5a66fd6314e43a28696e7faec4eb1b6` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:19.858186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5a66fd6314e43a28696e7faec4eb1b6)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:19.860484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:19.875419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:19.876172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.877412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.877768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] PATH_INFO: `/v3/projects/17e8edbb491942b5bca48b6ed8f78e95` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:19.880291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17e8edbb491942b5bca48b6ed8f78e95)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:19.882683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:19.934553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5a66fd6314e43a28696e7faec4eb1b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.944899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5a66fd6314e43a28696e7faec4eb1b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.945938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5a66fd6314e43a28696e7faec4eb1b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.947057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5a66fd6314e43a28696e7faec4eb1b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.960085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17e8edbb491942b5bca48b6ed8f78e95', 'request_id': 'req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.961886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4da80846-c9fc-4190-a8fe-246b0b21b722 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5a66fd6314e43a28696e7faec4eb1b6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.971750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17e8edbb491942b5bca48b6ed8f78e95', 'request_id': 'req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.982340 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17e8edbb491942b5bca48b6ed8f78e95', 'request_id': 'req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:19.993384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1261/2506] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:19 2022] DELETE /identity/v3/users/e5a66fd6314e43a28696e7faec4eb1b6 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:19.995392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:19.995678 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:19.995959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.002729 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] The token cache is being invalidate because project 17e8edbb491942b5bca48b6ed8f78e95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:20.003032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:20.003732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-bd0e90cc-ec7e-4910-b18e-eb26d8a962f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:20.005941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1246/2507] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:19 2022] DELETE /identity/v3/projects/17e8edbb491942b5bca48b6ed8f78e95 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:20.011919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2884f-4964-4bf5-84c4-173ba1ebb042 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.012423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2884f-4964-4bf5-84c4-173ba1ebb042 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.012712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2884f-4964-4bf5-84c4-173ba1ebb042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.028767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:20.030098 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.047967 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-85b2884f-4964-4bf5-84c4-173ba1ebb042 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:20.049204 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-85b2884f-4964-4bf5-84c4-173ba1ebb042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.069156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:20.121266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1262/2508] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:20.132533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.134223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.166341 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1247/2509] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:20.168828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.169173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.169578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.188770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:20.190740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:20.191920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.201430 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1248/2510] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:20.204792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9958fa3f-0c8d-4060-ae95-7f6313363ee3 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.205387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.209717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.210619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.210972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.212039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.215679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634685ac-16f3-44fb-ac53-084d0d644e55 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.221360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.222173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.222758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.222864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.222946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.223412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:20.224903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.225028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.226151 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.237480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1263/2511] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:20.240342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1249/2512] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:04:20.244340 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.254824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d0dc5f-195a-43fc-b1a4-e7005acc579e service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.260013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.260827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.261115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.261938 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.262624 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.264522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.299496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1264/2513] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:20 2022] POST /identity/v3/projects => generated 373 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:20.303798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.314490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13551cef-82ac-4f55-a7a5-568facdb8530 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.383970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.384698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.384941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.385356 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.386070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.388117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.410598 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1250/2514] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:20 2022] POST /identity/v3/users => generated 287 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:20.414646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.424796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b9f5f-afb6-41de-8873-5b2442a9f4ae admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.429447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.430079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.430359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.430630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.431265 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.432842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.443312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1265/2515] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:20.448492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.459872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb8117d-2174-4325-b705-582d3c0a6fd6 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.465317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.466192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.466383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.466619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.467300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.469251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbc0a82-3d6e-4bee-ad67-f514e196cdf2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.480852 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1251/2516] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:20.485663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.495986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c0736-29f0-4559-80ea-a2407c2ba463 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.500700 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.501376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.501652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.501926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] PATH_INFO: `/v3/projects/fc436cb987344c159e565353054e9539/users/8279ddc694824ebc8185f50fafbeec76/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.517459 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc436cb987344c159e565353054e9539, user_id=8279ddc694824ebc8185f50fafbeec76, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.520138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.534686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1266/2517] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:20 2022] PUT /identity/v3/projects/fc436cb987344c159e565353054e9539/users/8279ddc694824ebc8185f50fafbeec76/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:20.539820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.540115 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.540393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.560616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:20.561861 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.625437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1252/2518] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:20.636255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.694665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e97f31-02d4-4364-8903-7b8ed1df375f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.699494 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.700129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.700403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.700674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.711295 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.712781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2fec78-ab03-4148-993d-20b69f5ab780 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.723607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1267/2519] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:04:20.923165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:20.924613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:20.939936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac74cf56-c292-4f63-9426-23eb09c9a587 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:20.946287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:20.947042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:20.947670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:20.947670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] PATH_INFO: `/v3/projects/cf9538379ffd44ee9a3765f8dea19810` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:20.949274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf9538379ffd44ee9a3765f8dea19810)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:20.951528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:20.985420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:21.028545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf9538379ffd44ee9a3765f8dea19810', 'request_id': 'req-66089c2d-cd78-40c5-8e0d-1162facdfb91'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:21.035892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf9538379ffd44ee9a3765f8dea19810', 'request_id': 'req-66089c2d-cd78-40c5-8e0d-1162facdfb91'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:21.044353 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf9538379ffd44ee9a3765f8dea19810', 'request_id': 'req-66089c2d-cd78-40c5-8e0d-1162facdfb91'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:21.066155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] The token cache is being invalidate because project cf9538379ffd44ee9a3765f8dea19810 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:21.066538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:21.067955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-66089c2d-cd78-40c5-8e0d-1162facdfb91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:21.069782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1253/2520] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:20 2022] DELETE /identity/v3/projects/cf9538379ffd44ee9a3765f8dea19810 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:21.075373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.075657 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.075926 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.101366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:21.102777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.172486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1268/2521] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:21.181349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.181349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.181615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.203738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:21.204808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.214816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1254/2522] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:21.219546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.230294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08457fe-0b99-40f5-9a12-263f216b1d15 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.236104 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.236755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.237036 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.237311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.237968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:21.238311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.240586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.261686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1269/2523] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:04:21.265463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.277612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4c851b-a431-43c3-a4a6-256c4cfb529d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.283538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.284190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.284473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.284805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.285522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.288166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.322088 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1255/2524] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/projects => generated 383 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:21.326292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.337667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089fbcce-2786-4d97-b63d-646004d629bd admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.353946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:21.406301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.406930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.407210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.407485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.408143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.411577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.456309 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1270/2525] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:21.461789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.473549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38917e2-c5de-417d-ac49-06fe13e90c9d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.480344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.481167 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.481453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.481727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.483050 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.485344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475a1863-c29d-4e14-afe0-f426d32f5d10 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.505412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1256/2526] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:21.509321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.530991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf50720-ba5a-4999-9e2f-6b4eee90b118 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.538761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.539124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.539415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.539694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.539806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.540233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.540621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.541370 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.543283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.558461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1271/2527] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:21.562133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.578346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] MFA Rules not processed for user `8279ddc694824ebc8185f50fafbeec76`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:21.581201 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4271fa95-bed8-4231-b42c-51a4120afdb4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.581742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.592570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.593488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.593759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.594134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] PATH_INFO: `/v3/projects/27b47aa9a61b4898a9af0062896cc3da/users/859315e3352843fd801ddc6568729e56/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.625929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27b47aa9a61b4898a9af0062896cc3da, user_id=859315e3352843fd801ddc6568729e56, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.629171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e82d37-0f9c-4b7c-aea6-24867fc5ab22 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.660713 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1272/2528] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:21 2022] PUT /identity/v3/projects/27b47aa9a61b4898a9af0062896cc3da/users/859315e3352843fd801ddc6568729e56/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:04:21.664306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.664707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.666106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.695123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:21.696803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.728665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1257/2529] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:21.731127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.732661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.812396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f466d462-7303-4e4c-8816-5ed3a56b169a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.818818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1273/2530] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:21.819255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:21.827458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:21.830113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:21.900478 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a98223e-fcd0-4dc5-944b-8976bd19aa91 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:21.900911 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.902558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.902847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.903124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.906691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:21.907394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:21.907741 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:21.908092 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:21.919680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:21.921396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43bbe41-85b4-44dc-b86b-93a311eedb56 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:21.944633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1274/2531] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:04:22.000657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:22.002439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:22.034364 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1258/2532] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:04:24.691333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d6af5-cf83-43d0-a95a-1c1105b17c6a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:24.691860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d6af5-cf83-43d0-a95a-1c1105b17c6a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:24.691860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d6af5-cf83-43d0-a95a-1c1105b17c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:24.727607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-463d6af5-cf83-43d0-a95a-1c1105b17c6a None None] MFA Rules not processed for user `8279ddc694824ebc8185f50fafbeec76`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:24.729129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-463d6af5-cf83-43d0-a95a-1c1105b17c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:24.751146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1275/2533] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:26.026120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:26.030835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:26.081369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17791b66-eb7b-4ccf-ac06-b2bd61b03c1c service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:26.099468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:26.100137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:26.100412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:26.100889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] PATH_INFO: `/v3/users/3a7bbabb9e0f46eca84c8e83022431de` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:26.103969 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a7bbabb9e0f46eca84c8e83022431de)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:26.105922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:26.199727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a7bbabb9e0f46eca84c8e83022431de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:26.201005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a7bbabb9e0f46eca84c8e83022431de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:26.214415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a7bbabb9e0f46eca84c8e83022431de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:26.215258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a7bbabb9e0f46eca84c8e83022431de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:26.225777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a7bbabb9e0f46eca84c8e83022431de'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:26.282351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1259/2534] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:26 2022] DELETE /identity/v3/users/3a7bbabb9e0f46eca84c8e83022431de => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:26.287469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:26.287959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:26.289269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:26.333769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:26.336851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:26.346350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:26.454394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1276/2535] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:26.468865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:26.470453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:26.543861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:26.546506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:26.556765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:26.578828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f8912-c898-4a9a-9540-9a9778b94786 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:26.583064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:26.583814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:26.584099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:26.584768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:26.598872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:26.600706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:26.616257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1260/2536] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:26.661826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee1deb7-2860-41f4-b30f-58b619bdbf1f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:26.668482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:26.668729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:26.669101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:26.669507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:26.671473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:26.689508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:26.691599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9820ecbc-63b5-404a-9fa5-a5d45fec8aa5 service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:26.709147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1277/2537] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:27.284649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.293136 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:27.312191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46a6d4-9330-4646-be7c-730c6ff36739 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.317633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.318377 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.318826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.320622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] PATH_INFO: `/v3/projects/9205499b83d14ee2b0baba543b3e19b4` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.325475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9205499b83d14ee2b0baba543b3e19b4)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.327707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.460827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9205499b83d14ee2b0baba543b3e19b4', 'request_id': 'req-1e8034e5-8edf-4355-a29c-75ef18e2d60e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:27.469383 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9205499b83d14ee2b0baba543b3e19b4', 'request_id': 'req-1e8034e5-8edf-4355-a29c-75ef18e2d60e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:27.481863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9205499b83d14ee2b0baba543b3e19b4', 'request_id': 'req-1e8034e5-8edf-4355-a29c-75ef18e2d60e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:27.510062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] The token cache is being invalidate because project 9205499b83d14ee2b0baba543b3e19b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:27.510062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:27.510565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-1e8034e5-8edf-4355-a29c-75ef18e2d60e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:27.515999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1261/2538] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:27 2022] DELETE /identity/v3/projects/9205499b83d14ee2b0baba543b3e19b4 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:27.517659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b65081-641a-4edb-a035-8334c7e4d609 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.518189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b65081-641a-4edb-a035-8334c7e4d609 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.518285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b65081-641a-4edb-a035-8334c7e4d609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.540482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-95b65081-641a-4edb-a035-8334c7e4d609 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:27.541806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-95b65081-641a-4edb-a035-8334c7e4d609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:27.587667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.587955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.588225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.626491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] MFA Rules not processed for user `859315e3352843fd801ddc6568729e56`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:27.627735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:27.640290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1278/2539] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:27.649555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.649855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.650170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.673446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:27.674760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:27.685811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1279/2540] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:27.690023 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.700357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d42f5-f220-4b42-bd7c-2c47ec513cf9 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.705719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.706651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.708384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.708824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.709658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:04:27.710735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.712811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.733170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1262/2541] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:04:27.739231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1280/2542] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:04:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:04:27.739521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.739521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:27.741870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.756988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9de3d-5882-4aeb-be8d-fa02b40a1afc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.760120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe794b-232f-4d01-a90a-17e8208cf99f None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.765540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.766071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.766459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.766816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.767149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.767442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.767917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.768036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.768441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.769334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.770919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.772718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.799755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:27.810326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1281/2543] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/projects => generated 399 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:27.811802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1263/2544] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/projects => generated 383 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:04:27.815816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.816611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:27.873178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:27.924156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bae043-e66d-432c-a61d-8b1a9ba0be0b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.933324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9657dcd2-f446-4d2b-82f2-95a1e6134918 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:27.936319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.937177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.937888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.937888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.938714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.943309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.958787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:27.958787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:27.959180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:27.959343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:27.964279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:27.966360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:27.977761 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1264/2545] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/users => generated 298 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:28.005114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.015605 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1282/2546] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:04:27 2022] POST /identity/v3/users => generated 300 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:04:28.019762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.035034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befb00e4-41c7-4b44-8fe9-84674eba3d6e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.051128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b1d59-2910-4b31-9a9c-4407af84f97e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.053624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.054781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.055063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.055338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.063654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.070884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.074239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.075446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.075742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.076008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.076677 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.082487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.096420 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1265/2547] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:28.100534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.105730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1283/2548] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:28.110293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.115544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ac060-d359-4fd7-8afe-40890c002980 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.120454 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.121184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.121463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.121737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] PATH_INFO: `/v3/projects/153f5a4dc45649009721ffd9d22bd7d5/users/8ac35e78eb1c4ab8bae1d1934514fb83/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.134514 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4025cd-74f8-457f-9e5b-9e23e7e02f2f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.139721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.140420 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.140794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.141141 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.142057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.143825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.158897 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1284/2549] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:04:28.162862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.178071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198b2122-a078-44db-928d-80870ad330ea admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.182302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=153f5a4dc45649009721ffd9d22bd7d5, user_id=8ac35e78eb1c4ab8bae1d1934514fb83, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.185200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.190294 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.191297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.191594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.191884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] PATH_INFO: `/v3/projects/ea4fde1b333e44e0adae3e98c26d83be/users/e3228cbd40394614b249829fb900cf86/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.205893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1266/2550] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:28 2022] PUT /identity/v3/projects/153f5a4dc45649009721ffd9d22bd7d5/users/8ac35e78eb1c4ab8bae1d1934514fb83/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:28.210912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.220552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea4fde1b333e44e0adae3e98c26d83be, user_id=e3228cbd40394614b249829fb900cf86, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.225491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494d4ba-de83-4a45-86cc-29726ad7fbf8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.252647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1285/2551] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:28 2022] PUT /identity/v3/projects/ea4fde1b333e44e0adae3e98c26d83be/users/e3228cbd40394614b249829fb900cf86/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:28.270774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.271042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.271367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.319452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58caab23-d7ab-4050-9e63-061838556c90 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.343193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] MFA Rules not processed for user `e3228cbd40394614b249829fb900cf86`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:28.344440 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:28.463912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1286/2552] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:28 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:28.464372 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.465074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.465404 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.465771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.466622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.475804 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.476246 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.478231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:28.490027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1267/2553] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:04:28.492602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:28.527027 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6eb44f-9e19-461d-a09c-75c564dbfc99 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.532504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.533177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.533456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.534658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] PATH_INFO: `/v3/projects/153f5a4dc45649009721ffd9d22bd7d5/users/8ac35e78eb1c4ab8bae1d1934514fb83/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.535751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] RBAC: Authorizing `identity:create_grant(project_id=153f5a4dc45649009721ffd9d22bd7d5, user_id=8ac35e78eb1c4ab8bae1d1934514fb83, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.538964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4871dd1d-113e-424c-a0e3-d30416eab164 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.550145 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:28.553343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1268/2554] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:04:28 2022] PUT /identity/v3/projects/153f5a4dc45649009721ffd9d22bd7d5/users/8ac35e78eb1c4ab8bae1d1934514fb83/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:28.595528 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d59623-75c3-4cb0-a739-06e6a3147a70 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:28.683279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:28.684325 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:28.684990 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:28.685702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:28.785113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:28.786882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:28.802550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1287/2555] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:28 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:04:33.659342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acbb67-df4d-4768-a8fe-cf38043cebb3 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:33.659974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acbb67-df4d-4768-a8fe-cf38043cebb3 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:33.659974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80acbb67-df4d-4768-a8fe-cf38043cebb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:33.688155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:33.692298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-80acbb67-df4d-4768-a8fe-cf38043cebb3 None None] MFA Rules not processed for user `8ac35e78eb1c4ab8bae1d1934514fb83`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:33.694847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-80acbb67-df4d-4768-a8fe-cf38043cebb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:33.798654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1269/2556] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:33 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:33.812795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:33.814264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:33.896978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d8250-2922-4a31-a9f9-8938dca28289 service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:33.900773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:33.901493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:33.901920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:33.902273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:33.975504 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:33.980585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:33.982888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:33.996801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1288/2557] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:34.173215 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:34.174279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:34.174570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:34.235559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:34.236281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:34.340978 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1270/2558] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:04:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:34.358041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:34.359864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:34.446063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206b6f4-7d2a-4a35-a90f-70c5a9584973 service glance] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:34.451138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:34.451833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:34.452175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:34.452513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:34.461858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:34.464206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca4f96d-dd87-4cc9-b87d-ceaafbcafca4 service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:34.473236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1289/2559] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:04:42.751373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:42.840807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90271d8-3d98-4bc4-93b2-4629c40dffde None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:42.845400 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:42.846691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:42.846691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:42.846691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] PATH_INFO: `/v3/users/e3228cbd40394614b249829fb900cf86` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:42.848510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3228cbd40394614b249829fb900cf86)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:42.850465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:42.904095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:42.907434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3228cbd40394614b249829fb900cf86'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:42.908488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3228cbd40394614b249829fb900cf86'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:42.922018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3228cbd40394614b249829fb900cf86'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:42.923173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3228cbd40394614b249829fb900cf86'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:42.933925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3228cbd40394614b249829fb900cf86'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:42.957033 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1271/2560] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:04:42 2022] DELETE /identity/v3/users/e3228cbd40394614b249829fb900cf86 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:42.961480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:42.962050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:42.962436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:42.986887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:42.988235 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:43.069412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1290/2561] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:43.079350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:43.080894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:43.151810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e507d-cae4-484b-bbe4-593069d3ad7e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:43.155867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:43.156503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:43.156789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:43.157058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:43.168508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:43.170164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be547eab-d70f-4bf5-b0c8-498f9f9269ed service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:43.181724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1272/2562] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:04:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:04:43.527239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:43.528706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:43.545015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707ebeb6-3628-469a-8af0-ef7aed2718d5 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:43.551948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:43.552695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:43.553035 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:43.553316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] PATH_INFO: `/v3/projects/ea4fde1b333e44e0adae3e98c26d83be` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:43.554536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea4fde1b333e44e0adae3e98c26d83be)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:43.556468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:43.615345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea4fde1b333e44e0adae3e98c26d83be', 'request_id': 'req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:43.624489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea4fde1b333e44e0adae3e98c26d83be', 'request_id': 'req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:43.632357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea4fde1b333e44e0adae3e98c26d83be', 'request_id': 'req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:43.653999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] The token cache is being invalidate because project ea4fde1b333e44e0adae3e98c26d83be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:04:43.654336 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:43.655032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-66b9f4cc-90d7-43c8-a938-d6ec2cedc6a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:04:43.656802 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1291/2563] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:04:43 2022] DELETE /identity/v3/projects/ea4fde1b333e44e0adae3e98c26d83be => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:04:45.114069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:45.114678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:45.114678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:45.133896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] MFA Rules not processed for user `859315e3352843fd801ddc6568729e56`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:45.135133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:45.189609 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1273/2564] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:04:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:48.586769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:48.587292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:48.587543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:48.608072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:04:48.609299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:48.614089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:48.676730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1292/2565] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:04:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:04:48.689295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:04:48.690717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:04:48.771584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0b49c8-fc2e-4bc2-a3b0-8c843ecc736d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:04:48.777318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:04:48.777943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:04:48.778274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:04:48.778552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:04:48.799976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:04:48.801854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fbe2f2-9114-4263-b904-519ff8deaef0 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:04:48.807463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:04:48.820889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1274/2566] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:04:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:04:58.670146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=21 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:05:15.795079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:15.796673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:15.861519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a7d275-5e1a-4e46-9368-41897bef1f7d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:15.865992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:15.866638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:15.866917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:15.867190 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] PATH_INFO: `/v3/users/777461af3f1e4c218ac8278a132698bd` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:15.868324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=777461af3f1e4c218ac8278a132698bd)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:15.870259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:15.915622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '777461af3f1e4c218ac8278a132698bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:15.923318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '777461af3f1e4c218ac8278a132698bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:15.923936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '777461af3f1e4c218ac8278a132698bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:15.924550 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '777461af3f1e4c218ac8278a132698bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:15.931380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '777461af3f1e4c218ac8278a132698bd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:15.948940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1293/2567] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:05:15 2022] DELETE /identity/v3/users/777461af3f1e4c218ac8278a132698bd => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:05:15.953447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:15.953966 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:15.954125 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:15.972650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:15.973849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:16.033905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1275/2568] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:05:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:16.043972 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:16.045369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:16.101254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697f365c-44d3-400e-9206-27aa0870ce10 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:16.105408 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:16.106030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:16.106311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:16.106587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:16.115638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:16.117058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60f1a62-3b47-42f5-ab48-42cdf550b686 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:16.126380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1294/2569] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:05:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:05:16.489864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:16.491366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:16.492014 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a3dcde-b295-48e5-9975-ba69770827e8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:16.492303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a3dcde-b295-48e5-9975-ba69770827e8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:16.492581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a3dcde-b295-48e5-9975-ba69770827e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:16.504291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f95e4d8-b77b-44b6-8336-3bc69d6f57ae None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:16.518081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:16.518737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:16.519013 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:16.519283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] PATH_INFO: `/v3/projects/36a584cd24cd42179665ad2d6b3eea46` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:16.520432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36a584cd24cd42179665ad2d6b3eea46)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:16.522438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:16.532867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-64a3dcde-b295-48e5-9975-ba69770827e8 None None] MFA Rules not processed for user `8279ddc694824ebc8185f50fafbeec76`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:16.534160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-64a3dcde-b295-48e5-9975-ba69770827e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:16.554947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:05:16.603029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36a584cd24cd42179665ad2d6b3eea46', 'request_id': 'req-8638fe58-533e-4d65-b561-ee671d193815'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:16.608171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1295/2570] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:05:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:16.611016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36a584cd24cd42179665ad2d6b3eea46', 'request_id': 'req-8638fe58-533e-4d65-b561-ee671d193815'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:16.618479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36a584cd24cd42179665ad2d6b3eea46', 'request_id': 'req-8638fe58-533e-4d65-b561-ee671d193815'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:16.632533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] The token cache is being invalidate because project 36a584cd24cd42179665ad2d6b3eea46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:05:16.632843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:16.633460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:05:16.635123 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1276/2571] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:05:16 2022] DELETE /identity/v3/projects/36a584cd24cd42179665ad2d6b3eea46 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:05:26.599999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=15 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:05:26.629460 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:05:34.418664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:34.419859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:34.419859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:34.440969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:34.441874 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:34.510045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1296/2572] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:05:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:34.537202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:34.539180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:34.629883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638fe58-533e-4d65-b561-ee671d193815 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:34.638032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:34.638798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:34.639077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:34.639350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:34.654488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:34.656100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88583ab2-7d1b-4df7-b9a8-dd6010da56e8 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:34.665518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1277/2573] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:05:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:05:35.086988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:35.089302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:35.161944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8125a623-e42e-4fa5-b838-ff31aedc9e5c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:35.165773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:35.166436 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:35.166719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:35.166997 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:35.168762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:35.242452 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:35.244619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:35.257375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1297/2574] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:05:35 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:05:43.922402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836e010-2b32-4afe-8ff5-c09102d53c2a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:43.923486 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836e010-2b32-4afe-8ff5-c09102d53c2a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:43.924053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836e010-2b32-4afe-8ff5-c09102d53c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:43.951559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-0836e010-2b32-4afe-8ff5-c09102d53c2a None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:43.953706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0836e010-2b32-4afe-8ff5-c09102d53c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:43.968562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1278/2575] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:05:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:05:43.976673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:43.991899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf3646-c63f-408e-8968-396175f4cd2d service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:43.998541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:43.999532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:44.000043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:44.000537 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:44.013502 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:44.016548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2586f0-6203-4f85-8d4b-eaffec2d7f9b service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:44.029933 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1298/2576] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:05:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 11:05:51.971018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:51.972327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:51.972791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:51.974601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1279/2577] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:05:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:05:51.981769 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcd90a-9d33-4a10-96fd-cb2e9dc2a9d2 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:51.982725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcd90a-9d33-4a10-96fd-cb2e9dc2a9d2 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:51.983161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fcd90a-9d33-4a10-96fd-cb2e9dc2a9d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.007013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e2fcd90a-9d33-4a10-96fd-cb2e9dc2a9d2 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:52.008248 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e2fcd90a-9d33-4a10-96fd-cb2e9dc2a9d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:52.070068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:05:52.114664 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1299/2578] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:05:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:52.127036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:52.128842 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:52.206601 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44b43b6-885e-47ab-87fb-7e6865b6de1b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:52.210513 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:52.211371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.211722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.212068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.225399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:52.227154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68186ce-2b96-4e55-bb76-440bbf82bdfb service glance] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:52.239006 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1280/2579] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:05:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:05:52.326540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.326928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.327323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] PATH_INFO: `/` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.329552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1300/2580] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:05:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:05:52.334567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f02ac3-58f0-435c-93a0-2562a926f397 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.334864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f02ac3-58f0-435c-93a0-2562a926f397 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.335137 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f02ac3-58f0-435c-93a0-2562a926f397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.365899 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f6f02ac3-58f0-435c-93a0-2562a926f397 None None] MFA Rules not processed for user `0287d4acc8f548c482ad40bd61b1081d`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:52.367812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f6f02ac3-58f0-435c-93a0-2562a926f397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:52.376471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1281/2581] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:05:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:52.389541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:52.391552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:52.405728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c71fba-897a-4b1f-91db-6b4e06e55a21 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:52.409339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:52.409961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.410273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.410549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.419363 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:52.420822 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52cb986-a8b4-4a76-8ec2-a0770e1dc696 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:52.442013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1301/2582] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:05:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:05:52.520350 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb2c48-25a2-4127-98bc-06432572b48b None None] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.520708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb2c48-25a2-4127-98bc-06432572b48b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.520950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb2c48-25a2-4127-98bc-06432572b48b None None] PATH_INFO: `/` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.522330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1282/2583] 10.222.0.46 () {54 vars in 921 bytes} [Fri Oct 28 11:05:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 11:05:52.528046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ac0f8d-2509-44be-ad60-4038bf773541 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.528333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ac0f8d-2509-44be-ad60-4038bf773541 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.528610 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ac0f8d-2509-44be-ad60-4038bf773541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.551927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b6ac0f8d-2509-44be-ad60-4038bf773541 None None] MFA Rules not processed for user `6adb6b4eff2545c080d2eb3327b78d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:52.553082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b6ac0f8d-2509-44be-ad60-4038bf773541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:52.563064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1302/2584] 10.222.0.46 () {58 vars in 1018 bytes} [Fri Oct 28 11:05:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:52.573545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fb2c48-25a2-4127-98bc-06432572b48b None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:52.586243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fb2c48-25a2-4127-98bc-06432572b48b None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:52.590126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:52.591433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:52.591713 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:52.591984 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:52.605307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:52.606808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:52.618889 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1283/2585] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:05:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:05:55.387606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a2f3f-e837-4198-8095-a91468c91c8b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:55.388124 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a2f3f-e837-4198-8095-a91468c91c8b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:55.388218 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612a2f3f-e837-4198-8095-a91468c91c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:55.411398 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-612a2f3f-e837-4198-8095-a91468c91c8b None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:05:55.412837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-612a2f3f-e837-4198-8095-a91468c91c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:05:55.420254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1303/2586] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:05:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:05:55.435152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:05:55.447903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711ba27a-f5c6-4be1-a050-276495bd5584 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:05:55.453757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:05:55.454457 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:05:55.454686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:05:55.455139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:05:55.464085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:05:55.466275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:05:55.476710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1284/2587] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:05:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 11:06:00.899606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c61cf-4611-4481-9765-49497ef68636 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:00.900176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c61cf-4611-4481-9765-49497ef68636 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:00.900284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c61cf-4611-4481-9765-49497ef68636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:00.920911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-021c61cf-4611-4481-9765-49497ef68636 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:06:00.922178 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-021c61cf-4611-4481-9765-49497ef68636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:00.929395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1304/2588] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:06:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:06:00.941496 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:00.946938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=77 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:00.953491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ef3ef2-6a50-4b1f-a889-9f5402fe25aa service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:00.958550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:00.959146 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:00.959439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:00.959704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:00.968382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:00.969964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:00.979405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1285/2589] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:06:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:06:03.005144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c61cf-4611-4481-9765-49497ef68636 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:03.006895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-021c61cf-4611-4481-9765-49497ef68636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:03.080576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c61cf-4611-4481-9765-49497ef68636 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:03.090561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:03.091306 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:03.091663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:03.092013 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] PATH_INFO: `/v3/users/8279ddc694824ebc8185f50fafbeec76` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:03.093251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] RBAC: Authorizing `identity:delete_user(user_id=8279ddc694824ebc8185f50fafbeec76)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:03.095437 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:03.156909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8279ddc694824ebc8185f50fafbeec76'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.174498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8279ddc694824ebc8185f50fafbeec76'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.176432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8279ddc694824ebc8185f50fafbeec76'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.178275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8279ddc694824ebc8185f50fafbeec76'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.202893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8279ddc694824ebc8185f50fafbeec76'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.224908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1305/2590] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:06:03 2022] DELETE /identity/v3/users/8279ddc694824ebc8185f50fafbeec76 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:06:03.475813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:03.487209 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a648056-00cc-4bfe-b10b-b3e78dbd347b service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:03.543920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:03.544630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:03.544913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:03.545302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] PATH_INFO: `/v3/projects/fc436cb987344c159e565353054e9539` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:03.546583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc436cb987344c159e565353054e9539)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:03.549038 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:03.614361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc436cb987344c159e565353054e9539', 'request_id': 'req-7a6314cb-a967-4f7f-80e3-40cc408a8993'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.619808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc436cb987344c159e565353054e9539', 'request_id': 'req-7a6314cb-a967-4f7f-80e3-40cc408a8993'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.625929 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc436cb987344c159e565353054e9539', 'request_id': 'req-7a6314cb-a967-4f7f-80e3-40cc408a8993'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.639662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] The token cache is being invalidate because project fc436cb987344c159e565353054e9539 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:06:03.640227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.640898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:06:03.642467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1286/2591] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:06:03 2022] DELETE /identity/v3/projects/fc436cb987344c159e565353054e9539 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:06:13.221064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=80,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:13.635201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=56,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:17.912914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:17.914396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:17.969920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f33442-0276-4d97-9906-6fbf58a9762f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:18.016869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:18.017472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:18.017749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:18.018044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:18.019736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:18.088607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:18.090082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1d2baa-18af-4317-8113-b75957c39c7e service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:18.101600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1306/2592] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:19.845086 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:19.846592 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:19.907467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6314cb-a967-4f7f-80e3-40cc408a8993 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:19.911240 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:19.911846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:19.912124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:19.912397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:19.914152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:19.928636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:19.930179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:19.941275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1287/2593] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:26.083921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:26.084438 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:26.084642 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:26.105499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:06:26.106735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:26.161156 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1307/2594] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:06:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:06:26.174750 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:26.176194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:26.247631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89baa90-f47f-4def-8b33-42df27dbddf3 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:26.253212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:26.254300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:26.254939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:26.255216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:26.264574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:26.266185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:26.274709 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1288/2595] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:06:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:06:28.147133 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:28.148560 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:28.158293 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:28.164787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aef63a-8a5c-48c6-bd57-0923ad064058 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:28.170239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:28.170840 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:28.171119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:28.171393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:28.173058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:28.229633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:28.231252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:28.241011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1308/2596] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:28 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:28.691850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:28.693173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:28.708669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ce148-e78c-46ed-b103-f8a66b582c7f service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:28.713922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:28.714539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:28.714815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:28.715081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:28.716736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:28.777186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:28.778803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b502b7-bad5-4a74-8497-47b58b80e85c service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:28.787743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1289/2597] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:28 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:38.237389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=27 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:38.783536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=86 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:06:45.040660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:45.049909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a2c0ce-dcb5-4db4-b3b2-b218dd46ada4 service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:45.053445 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:45.054085 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:45.054367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:45.054641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:45.056333 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:45.066979 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:45.068425 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:45.076310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1309/2598] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:47.664938 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:47.665459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:47.665459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:47.683943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] MFA Rules not processed for user `347e262907da4a83a197aaf42cecf51d`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:06:47.685130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:47.692305 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1290/2599] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:06:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:06:52.618656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:06:52.632033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92e7c1-fc8b-4a77-934b-b2cc1ee2f2bb service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:06:52.640364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:06:52.640364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:06:52.640364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:06:52.640364 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:06:52.641050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:06:52.653821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:06:52.655998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89427a-1b32-42d2-8856-31fbf8a1ae2b service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:06:52.667967 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1310/2600] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:06:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:06:57.678507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:07:02.663037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=12 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:07:07.091036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:07:07.107085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7435cec-86c5-4ce0-b3f4-ba13ffaea005 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:07:07.114157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:07:07.115029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:07.115424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:07.115791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:07.117733 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:07.132591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:07:07.134555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:07:07.158176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1291/2601] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:07:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:07:08.729025 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:08.729617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:08.729617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:08.751738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] MFA Rules not processed for user `6b926f7161c0487a8c2b04ada3b374e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:07:08.753062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:08.816360 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1311/2602] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:07:08 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:07:09.140128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:07:09.141552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:09.152577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24717f5-bc06-4d20-8b6f-3f6d7265a407 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:07:09.156523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:07:09.157110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:09.157384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:09.157650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:09.159297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:09.168206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:07:09.169624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:07:09.178095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1292/2603] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:07:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:07:09.626823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:07:09.628279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:09.698471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe6d75c-046c-41ba-a067-d42f6a51f9e7 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:07:09.703936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16220401-07d3-4815-925a-3579be076645 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:07:09.704541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16220401-07d3-4815-925a-3579be076645 service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:09.704818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16220401-07d3-4815-925a-3579be076645 service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:09.705091 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16220401-07d3-4815-925a-3579be076645 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:09.706826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-16220401-07d3-4815-925a-3579be076645 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:09.717962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16220401-07d3-4815-925a-3579be076645 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:07:09.720224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16220401-07d3-4815-925a-3579be076645 service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:07:09.729903 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1312/2604] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:07:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:07:19.174681 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=12 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:07:19.725760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=57 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:07:23.618948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:07:23.634536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0e02ae-af5f-48b1-9bb1-1ea5d367defc service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:07:23.640589 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:07:23.641857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:23.642169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:23.642453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:23.644207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:23.663873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:07:23.665674 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:07:23.683663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1293/2605] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:07:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:07:33.236396 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:33.236886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:33.236994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:33.256555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] MFA Rules not processed for user `fc81231c965947ecaced8013076edafc`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:07:33.258049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:33.264765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1313/2606] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:07:33 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:07:33.564079 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:07:33.573317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2dd595-1e02-47d8-bf32-adf52109feef service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:07:33.576837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:07:33.577431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:07:33.577706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:07:33.578519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:07:33.580246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:07:33.589787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:07:33.591254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:07:33.599940 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1294/2607] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:07:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:07:43.249932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:07:43.596222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=12 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:07.020189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:07.029960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d9e78-20a9-4de9-bde1-aec1f1e8fa1b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:07.033491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:07.034128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:07.034410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:07.034688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:07.036399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:07.098735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:07.100197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:07.110011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1314/2608] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:08:10.300909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:10.302472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:10.316585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a1aa6-7b99-4a57-aaba-287347ed05c6 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:10.321153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:10.321783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.322111 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.322416 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] PATH_INFO: `/v3/users/347e262907da4a83a197aaf42cecf51d` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.323545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=347e262907da4a83a197aaf42cecf51d)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:10.325452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:10.371787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '347e262907da4a83a197aaf42cecf51d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.372428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '347e262907da4a83a197aaf42cecf51d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.379960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '347e262907da4a83a197aaf42cecf51d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.380631 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '347e262907da4a83a197aaf42cecf51d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.387469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-010802f8-7f17-4497-9a34-f74b341f9db6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '347e262907da4a83a197aaf42cecf51d'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.405205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1295/2609] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:08:10 2022] DELETE /identity/v3/users/347e262907da4a83a197aaf42cecf51d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:10.573361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:10.583959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84abd8b3-22e2-428e-b731-2f0361831265 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:10.641037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:10.641673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.642547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.642839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] PATH_INFO: `/v3/projects/f1f49235fbe64b79bfef2cf828993c56` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.643956 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1f49235fbe64b79bfef2cf828993c56)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:10.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:10.690998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1f49235fbe64b79bfef2cf828993c56', 'request_id': 'req-48bf5824-810e-4ff8-9cef-df9e489cc48f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.696656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1f49235fbe64b79bfef2cf828993c56', 'request_id': 'req-48bf5824-810e-4ff8-9cef-df9e489cc48f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.702731 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1f49235fbe64b79bfef2cf828993c56', 'request_id': 'req-48bf5824-810e-4ff8-9cef-df9e489cc48f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.716146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] The token cache is being invalidate because project f1f49235fbe64b79bfef2cf828993c56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:08:10.716456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.717073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48bf5824-810e-4ff8-9cef-df9e489cc48f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:10.718593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1315/2610] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:08:10 2022] DELETE /identity/v3/projects/f1f49235fbe64b79bfef2cf828993c56 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:08:10.723584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.723870 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.724138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.743550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:10.744743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:10.803378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1296/2611] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:10.811455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.811734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.812024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.832182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:10.833367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:10.841049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1316/2612] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:08:10.845176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:10.854841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bc5c3-8897-46d4-bc00-b7657573de11 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:10.859587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:10.860190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.860465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.861250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.862236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:08:10.862538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:10.864154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:10.880252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1297/2613] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:08:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:08:10.881559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:10.890699 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef7ca3c-1299-4f10-bee9-f9953f2a4e72 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:10.895460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:10.896052 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.896330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:10.896598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:10.897214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:10.899128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:10.914662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:10.928417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1317/2614] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:08:10 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:10.932660 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:10.942884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d36e6e-1f0d-4457-a1b9-fab3f61af431 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:10.998977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:10.999579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:10.999880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.000133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.000754 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:11.002759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:11.010654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:11.025358 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1298/2615] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:08:10 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:11.029357 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:11.040244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce417ff4-72ab-40bc-a7ec-49455cadb8c5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:11.044993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:11.045591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:11.045869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.046193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.046854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:11.048495 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:11.058312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1318/2616] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:08:11.062023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:11.071789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac2567-4f2e-4a95-9e05-d442f3b6d15c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:11.076276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:11.076864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:11.077140 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.077408 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.078083 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:11.080160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d6d898-ef44-46c8-9f3d-7e9064e04edb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:11.090980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1299/2617] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:08:11.095603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:11.105689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fd09a7-f046-4234-8418-f6794bc7d1c9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:11.110240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:11.110889 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:11.111195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.111724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] PATH_INFO: `/v3/projects/64177183598843a68fc7fbe8257d4881/users/a1f71865b51b4e8480932fffae985567/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.125864 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64177183598843a68fc7fbe8257d4881, user_id=a1f71865b51b4e8480932fffae985567, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:11.128508 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:11.140641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1319/2618] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:08:11 2022] PUT /identity/v3/projects/64177183598843a68fc7fbe8257d4881/users/a1f71865b51b4e8480932fffae985567/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:08:11.146068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e98c2-b964-4902-84ed-24afa2e68f74 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:11.146351 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e98c2-b964-4902-84ed-24afa2e68f74 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.146622 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e98c2-b964-4902-84ed-24afa2e68f74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.166579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-0f3e98c2-b964-4902-84ed-24afa2e68f74 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:11.167771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0f3e98c2-b964-4902-84ed-24afa2e68f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:11.225455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1300/2619] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:11.235074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:11.237069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:11.293049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dc0516-a06b-434d-9697-72ae3a467098 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:11.296674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:11.297273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:11.297549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:11.297819 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:11.306544 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:11.307955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:11.317638 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1320/2620] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:08:15.164041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f162eb1-4b32-4d7b-afe8-5c699ca5b049 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:15.164532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f162eb1-4b32-4d7b-afe8-5c699ca5b049 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:15.164629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f162eb1-4b32-4d7b-afe8-5c699ca5b049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:15.196762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9f162eb1-4b32-4d7b-afe8-5c699ca5b049 None None] MFA Rules not processed for user `a1f71865b51b4e8480932fffae985567`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:15.201659 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9f162eb1-4b32-4d7b-afe8-5c699ca5b049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:15.272023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1301/2621] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:15.287068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:15.288646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:15.374585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990b1200-d50f-4b5c-8b2e-9229f9e713f1 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:15.385485 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:15.385788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:15.386167 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:15.386522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:15.395076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:15.396641 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52df74c1-3438-4117-9555-6e82b2c01d3a service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:15.405339 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1321/2622] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:08:16.748414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:16.748948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:16.749045 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:16.769475 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] MFA Rules not processed for user `a1f71865b51b4e8480932fffae985567`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:16.770738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:16.777543 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1302/2623] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:08:25.401311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=77,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:26.748076 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:26.748635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:26.748635 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:26.763711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:INSERT=3,keystone:SELECT=69 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:26.768776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:26.770030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:26.828107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1322/2624] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:08:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:26.875343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:26.876784 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:26.947771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5f0764-4f60-4c24-831a-00e7b220f687 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:26.952676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:26.953283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:26.953559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:26.953827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:26.962229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:26.963785 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:26.972107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1303/2625] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:08:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:08:33.875237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:33.877175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:33.937492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:33.939057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:33.952992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064cf38e-e499-4f65-a5c8-dd3708bdf387 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:33.957581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e32a4e9-abea-4286-a68b-95e3ab9fb991 service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:33.959112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:33.959807 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:33.960090 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:33.960382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:33.962148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:33.962271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:33.962792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:33.963082 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:33.963352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] PATH_INFO: `/v3/users/859315e3352843fd801ddc6568729e56` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:33.965536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=859315e3352843fd801ddc6568729e56)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:33.967711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:33.975462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:33.977031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:33.988131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1323/2626] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:08:34.017468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '859315e3352843fd801ddc6568729e56'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.018189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '859315e3352843fd801ddc6568729e56'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.028905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '859315e3352843fd801ddc6568729e56'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.029698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '859315e3352843fd801ddc6568729e56'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.037621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '859315e3352843fd801ddc6568729e56'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.055535 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1304/2627] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:08:33 2022] DELETE /identity/v3/users/859315e3352843fd801ddc6568729e56 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:08:34.236390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:34.249373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c1416a-8f8b-4127-9b15-7c80062da7b9 service placement] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:34.308960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:34.309604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:34.309887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:34.310204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] PATH_INFO: `/v3/projects/27b47aa9a61b4898a9af0062896cc3da` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:34.311385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27b47aa9a61b4898a9af0062896cc3da)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:34.313992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:34.356882 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:34.368976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27b47aa9a61b4898a9af0062896cc3da', 'request_id': 'req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.374977 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27b47aa9a61b4898a9af0062896cc3da', 'request_id': 'req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.382386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27b47aa9a61b4898a9af0062896cc3da', 'request_id': 'req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.396596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] The token cache is being invalidate because project 27b47aa9a61b4898a9af0062896cc3da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:08:34.396885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.397609 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.399447 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1324/2628] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:08:34 2022] DELETE /identity/v3/projects/27b47aa9a61b4898a9af0062896cc3da => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:34.403585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:34.405044 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:34.419229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85513856-c7a1-4e98-92b0-bcdbaa6632b1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:34.480216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:34.480857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:34.481139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:34.481414 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] PATH_INFO: `/v3/users/8ac35e78eb1c4ab8bae1d1934514fb83` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:34.482573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ac35e78eb1c4ab8bae1d1934514fb83)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:34.484509 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:34.487000 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:34.533418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ac35e78eb1c4ab8bae1d1934514fb83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.533990 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ac35e78eb1c4ab8bae1d1934514fb83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.541437 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ac35e78eb1c4ab8bae1d1934514fb83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.542108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ac35e78eb1c4ab8bae1d1934514fb83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.549234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-926549d3-550d-4c7e-9519-686c41cd8da6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ac35e78eb1c4ab8bae1d1934514fb83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.567020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1305/2629] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:08:34 2022] DELETE /identity/v3/users/8ac35e78eb1c4ab8bae1d1934514fb83 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:34.737283 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:34.738800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:34.751962 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7734f1-dade-46f3-8b4a-3139a46fcfe6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:34.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:34.808800 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:34.809083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:34.809359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] PATH_INFO: `/v3/projects/153f5a4dc45649009721ffd9d22bd7d5` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:34.810497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=153f5a4dc45649009721ffd9d22bd7d5)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:34.812395 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:34.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '153f5a4dc45649009721ffd9d22bd7d5', 'request_id': 'req-3858bea3-44ea-4989-8f14-a026e4c8bd85'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.865050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '153f5a4dc45649009721ffd9d22bd7d5', 'request_id': 'req-3858bea3-44ea-4989-8f14-a026e4c8bd85'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.870987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '153f5a4dc45649009721ffd9d22bd7d5', 'request_id': 'req-3858bea3-44ea-4989-8f14-a026e4c8bd85'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.883599 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] The token cache is being invalidate because project 153f5a4dc45649009721ffd9d22bd7d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:08:34.884259 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.884487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-3858bea3-44ea-4989-8f14-a026e4c8bd85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:08:34.886010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1325/2630] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:08:34 2022] DELETE /identity/v3/projects/153f5a4dc45649009721ffd9d22bd7d5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:34.891119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:34.891458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:34.891730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:34.910935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:34.912117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:34.969721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1306/2631] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:35.033557 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.033843 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.034155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.054115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:35.055321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:35.063821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1326/2632] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:35.068366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.078177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd45036-be3d-4036-a6f5-548e007aec9d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.082791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.083382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.083658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.083928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.084567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:08:35.084866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.086521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.097757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1307/2633] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:08:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:08:35.102046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.111844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea8a03d-46b8-426a-bfd2-0be2c193a902 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.116422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.117017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.117299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.117573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.118245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.120139 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.147191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1327/2634] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:08:35 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:35.151265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.160979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab4d52-8d41-4050-924e-b1567e95d665 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.221699 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.222331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.222607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.222877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.223492 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.225407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.241338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:35.252385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.252571 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1308/2635] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:08:35 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:35.262491 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac5183-5311-46b6-b6d7-002eb5947610 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.267040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.267645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.267927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.268197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.268832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.270449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.280535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1328/2636] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:08:35.285081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.294930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd562c-7e38-4d35-8759-072d8911dd1b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.299551 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.300421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.300691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.301327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.302974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e7896-73d9-4b73-ba73-11c3890e7814 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.314945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1309/2637] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:08:35.318842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.329955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226e76d-f621-4ba0-a7d3-150a22b91e97 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.334649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.335302 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.335582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.335858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] PATH_INFO: `/v3/projects/9edc5ff928624d87944ee071e578aa17/users/d4ade5932a324a4b8956b5377256ce7a/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.347509 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:35.350955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9edc5ff928624d87944ee071e578aa17, user_id=d4ade5932a324a4b8956b5377256ce7a, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.353824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.368371 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1329/2638] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:08:35 2022] PUT /identity/v3/projects/9edc5ff928624d87944ee071e578aa17/users/d4ade5932a324a4b8956b5377256ce7a/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:35.373530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df30fee-5172-4653-b511-00c4b360469d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.373813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df30fee-5172-4653-b511-00c4b360469d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.374122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df30fee-5172-4653-b511-00c4b360469d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.393555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-0df30fee-5172-4653-b511-00c4b360469d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:35.394799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0df30fee-5172-4653-b511-00c4b360469d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:35.451749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1310/2639] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:35.461852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:35.463559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:35.519569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34713d-5844-4f5d-938f-e62551f6513c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:35.523176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:35.523772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:35.524048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:35.524319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:35.533083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:35.534597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:35.543719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1330/2640] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:08:39.180318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.180771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.180771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.209428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] MFA Rules not processed for user `d4ade5932a324a4b8956b5377256ce7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:39.210588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:39.276508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1311/2641] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:39.281375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.291051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0a49-88d9-459e-b246-5b2605ee337e service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.295516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.296115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.296388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.296682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.297343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.299316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.326356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1331/2642] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:08:39 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:39.330936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.341608 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eba0537-dd1b-4ec7-ae9f-51c1f7b850b9 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.405339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.406507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.406850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.407128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.407775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.409722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.431788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1312/2643] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:08:39 2022] POST /identity/v3/users => generated 291 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:08:39.436134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.447885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab364109-d73e-46a4-ad1c-7e8632775e07 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.453335 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.454015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.454419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.454695 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.455417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.457213 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.469280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1332/2644] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:08:39.473531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.483955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab909ad-34ea-41c7-a830-a21170b6aadb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.489717 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.490442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.490722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.490999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] PATH_INFO: `/v3/projects/f998803074924d57b1e6ba81c117cdfd/users/9bc019301fea41098084e6d2ce86fc51/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.493298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:39.510324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f998803074924d57b1e6ba81c117cdfd, user_id=9bc019301fea41098084e6d2ce86fc51, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.513015 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.527795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1313/2645] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:08:39 2022] PUT /identity/v3/projects/f998803074924d57b1e6ba81c117cdfd/users/9bc019301fea41098084e6d2ce86fc51/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:39.532053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.543102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674a6bc-9277-4708-a31f-9c99a16495b5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.622112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.622739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.623017 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.623284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.623921 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.625592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3193f3-b42c-49fa-9d95-4ab293f540b3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.636298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1333/2646] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:08:39.640560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:39.651748 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4ec19c-7346-45c5-9327-c705579a4c87 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:39.656405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:39.657711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:39.658007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:39.658285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] PATH_INFO: `/v3/projects/f998803074924d57b1e6ba81c117cdfd/users/9bc019301fea41098084e6d2ce86fc51/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:39.660206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f998803074924d57b1e6ba81c117cdfd, user_id=9bc019301fea41098084e6d2ce86fc51, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:39.662895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:39.675588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1314/2647] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:08:39 2022] PUT /identity/v3/projects/f998803074924d57b1e6ba81c117cdfd/users/9bc019301fea41098084e6d2ce86fc51/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:08:44.212932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacc2045-f9df-4585-bb28-9b39eb51df73 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:44.213390 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacc2045-f9df-4585-bb28-9b39eb51df73 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:44.213486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacc2045-f9df-4585-bb28-9b39eb51df73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:44.246878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-dacc2045-f9df-4585-bb28-9b39eb51df73 None None] MFA Rules not processed for user `9bc019301fea41098084e6d2ce86fc51`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:44.248500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-dacc2045-f9df-4585-bb28-9b39eb51df73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:44.300662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:44.326859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1334/2648] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:44 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:44.346060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:08:44.348975 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:44.487781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038344f-8288-489a-b67f-d8c0dbf892d7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:08:44.494418 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:08:44.495334 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:44.495753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:44.496378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:44.567519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:08:44.569533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:08:44.579803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1315/2649] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:08:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:08:45.545401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920277f-d98d-4096-918a-f86f15b61b6a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:08:45.545919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920277f-d98d-4096-918a-f86f15b61b6a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:08:45.546043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920277f-d98d-4096-918a-f86f15b61b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:08:45.565771 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a920277f-d98d-4096-918a-f86f15b61b6a None None] MFA Rules not processed for user `d4ade5932a324a4b8956b5377256ce7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:08:45.567046 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a920277f-d98d-4096-918a-f86f15b61b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:08:45.573867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1335/2650] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:08:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:08:54.575362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=68,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:08:55.559718 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=11 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:09:25.587960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:09:25.589401 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:25.672337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5e774-6d2c-4450-a9dc-8d78995af535 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:09:25.676638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:09:25.677272 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:25.677546 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:25.677816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:25.707251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:09:25.708669 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:09:25.717853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1316/2651] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:09:25 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:09:29.443414 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:29.443940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:29.444042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:29.475480 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:09:29.476688 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:29.552148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1336/2652] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:09:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:09:29.604283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:09:29.605779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:29.680095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657e7007-9b10-4832-9d3c-2e9c72a269a8 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:09:29.685188 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:09:29.685736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:29.686040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:29.686314 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:29.695211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:09:29.696795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:09:29.704796 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1317/2653] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:09:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 11:09:32.407026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:09:32.408597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:32.474127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe11327-b831-402a-ab57-16017bca4e60 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:09:32.477876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:09:32.478598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:32.478945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:32.479288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:32.481117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:32.557847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:09:32.559446 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65694a17-7d5e-455a-a089-a3fcaa57f9cc service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:09:32.569723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1337/2654] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:09:32 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:09:32.984746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:09:32.987892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:33.037039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=100 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:09:33.088840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a72c6f-dab9-40b4-82f3-b27c2e6d8377 service swift] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:09:33.094295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:09:33.095477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:33.095756 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:33.096042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:33.097884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:33.109936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:09:33.112387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a82020-70ba-4dac-ae98-6694e78d4b06 service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:09:33.123600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1318/2655] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:09:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:09:42.565857 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=80 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:09:42.814061 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:42.814344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:42.814650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:42.833558 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] MFA Rules not processed for user `6b926f7161c0487a8c2b04ada3b374e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:09:42.834772 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:42.841463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1338/2656] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:09:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:09:43.119542 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=21 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:09:47.717802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0c20d-c12d-4498-a54f-6377ac8952fa None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:47.718787 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0c20d-c12d-4498-a54f-6377ac8952fa None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:47.719067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0c20d-c12d-4498-a54f-6377ac8952fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:47.739360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-91c0c20d-c12d-4498-a54f-6377ac8952fa None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:09:47.740565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-91c0c20d-c12d-4498-a54f-6377ac8952fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:09:47.747664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1319/2657] 10.222.0.46 () {58 vars in 1009 bytes} [Fri Oct 28 11:09:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:09:47.760206 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:09:47.772466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55fb142-1efe-4c88-bb4b-785dcbdd7ebc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:09:47.777481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:09:47.778122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:09:47.778399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:09:47.778668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:09:47.787041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:09:47.788686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:09:47.797029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1339/2658] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:09:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:09:57.733617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:09:57.793795 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=9 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:10:12.888673 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:10:12.889338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:10:12.889338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:10:12.909753 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] MFA Rules not processed for user `3c56f73fdd7742d6a336f41ff9da3578`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:10:12.911185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:10:12.918763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1320/2659] 10.222.0.46 () {58 vars in 967 bytes} [Fri Oct 28 11:10:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:10:12.931188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:10:12.941735 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3524447b-662a-41ab-b8fe-5588f2ba5518 service swift] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:10:12.947409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0449fafc849145c8b752a0b028ad2679', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:10:12.948120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:10:12.948461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:10:12.948823 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:10:12.957103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:10:12.959343 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:10:12.967598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1340/2660] 10.222.0.46 () {60 vars in 1384 bytes} [Fri Oct 28 11:10:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 11:10:20.221521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:10:20.241934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8875976-9b41-430a-89dc-70eb009b2a8c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:10:20.245554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:10:20.246208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:10:20.246489 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:10:20.246762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:10:20.248481 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:10:20.311758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:10:20.313192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:10:20.323450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1321/2661] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:10:20 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:10:22.964313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:10:30.318506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=37 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:20.014776 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:20.023986 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e38d2-bfac-41ab-80bb-66069dae95fa service swift] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:20.027597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:20.028189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:20.028468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:20.028739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:20.030472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:20.115083 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:20.116512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:20.135482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1341/2662] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:22.983176 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:22.984791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:23.000896 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e233e325-4efb-4d10-a611-9dda784b9ce7 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.005616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.006288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.006570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.006846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] PATH_INFO: `/v3/users/fc81231c965947ecaced8013076edafc` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.020819 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc81231c965947ecaced8013076edafc)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.022843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.067327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc81231c965947ecaced8013076edafc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.067972 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc81231c965947ecaced8013076edafc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.074906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc81231c965947ecaced8013076edafc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.075506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc81231c965947ecaced8013076edafc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.082605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-081a1820-73e1-4d1b-99c5-518c642f2405 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc81231c965947ecaced8013076edafc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.100098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1322/2663] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:11:22 2022] DELETE /identity/v3/users/fc81231c965947ecaced8013076edafc => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:11:23.277806 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.289442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431da7c5-bf5b-4e1b-9423-48ac1a5b78dc service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.346867 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.347496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.347775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.348050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] PATH_INFO: `/v3/projects/312745d2566343faa50fa7c6f06483f1` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.358100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] RBAC: Authorizing `identity:delete_project(project_id=312745d2566343faa50fa7c6f06483f1)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.360015 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.405080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '312745d2566343faa50fa7c6f06483f1', 'request_id': 'req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.410506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '312745d2566343faa50fa7c6f06483f1', 'request_id': 'req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.411575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:23.420205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '312745d2566343faa50fa7c6f06483f1', 'request_id': 'req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.432957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] The token cache is being invalidate because project 312745d2566343faa50fa7c6f06483f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:11:23.433245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.433930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-ddd444dd-ec9f-4501-ad9c-386cf63f4ffe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:11:23.435449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1342/2664] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:11:23 2022] DELETE /identity/v3/projects/312745d2566343faa50fa7c6f06483f1 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:11:23.440560 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.440849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.441124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.499638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:11:23.500853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:23.558767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1323/2665] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:11:23.566763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.567049 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.567322 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.587116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:11:23.588311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:23.596045 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1343/2666] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:11:23.600326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.609846 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09463bb7-1ac1-46e1-b557-9b979971e690 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.614642 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.615237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.615538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.615810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.616452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:11:23.616755 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.618890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.630181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1324/2667] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:11:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:11:23.634765 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.643716 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df1260f-9fb5-43b7-b6b7-398b443ad569 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.648303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.648939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.649183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.649453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.650119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.651989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.687068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1344/2668] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:11:23 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:11:23.691190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.701702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a116b20-81aa-4ed0-a256-b9e4c02bbc9d admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.713957 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:23.760904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.761520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.761802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.762105 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.762775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.764728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.786320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1325/2669] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:11:23 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:11:23.790963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.801142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f2d5d-c8c6-4728-92a4-5d6b94fe3107 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.806455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.807051 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.807332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.807606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.808243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.809826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.819598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1345/2670] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:11:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:11:23.824252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.834322 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b4acf-e407-4abb-8b0e-93a4903d4526 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.840591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.841537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.842042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.842508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.843594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.846468 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedf6973-1831-40a8-bfa6-44c3be439b82 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.859654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1326/2671] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:11:23 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:11:23.863847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:23.874596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fefc64-4a20-4eba-972b-bff4ace8b366 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:23.879197 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:23.879844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.880128 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.880403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] PATH_INFO: `/v3/projects/68345feb30fe454083122ecaf73ef747/users/ff4b7731f96643d0a24e63d31f5f6f00/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.894900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68345feb30fe454083122ecaf73ef747, user_id=ff4b7731f96643d0a24e63d31f5f6f00, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:23.897535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:23.910080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1346/2672] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:11:23 2022] PUT /identity/v3/projects/68345feb30fe454083122ecaf73ef747/users/ff4b7731f96643d0a24e63d31f5f6f00/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:11:23.915613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36497191-a5f3-43cc-8dd5-d5d4bec17c53 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:23.915994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36497191-a5f3-43cc-8dd5-d5d4bec17c53 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:23.916390 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36497191-a5f3-43cc-8dd5-d5d4bec17c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:23.936791 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-36497191-a5f3-43cc-8dd5-d5d4bec17c53 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:11:23.938072 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-36497191-a5f3-43cc-8dd5-d5d4bec17c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:23.996076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1327/2673] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:11:24.005768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:24.007471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:24.074463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9823e3d-e116-42f4-af5e-a826dc714ae3 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:24.078146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:24.078747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:24.079022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:24.079292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:24.088996 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:24.090441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:24.099674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1347/2674] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:28.030780 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e25fc-a792-42ae-8cf6-9a89b5f87881 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:28.031691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e25fc-a792-42ae-8cf6-9a89b5f87881 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:28.031970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0e25fc-a792-42ae-8cf6-9a89b5f87881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:28.062030 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4f0e25fc-a792-42ae-8cf6-9a89b5f87881 None None] MFA Rules not processed for user `ff4b7731f96643d0a24e63d31f5f6f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:11:28.063265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4f0e25fc-a792-42ae-8cf6-9a89b5f87881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:28.131908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1328/2675] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:11:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:11:28.145944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:28.147410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:28.219625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:28.258964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13046561-6ec1-4c62-9136-a53a2384ff22 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:28.266461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:28.271539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:28.271832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:28.272106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:28.283620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:28.285249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:28.294859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1348/2676] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:30.836011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:30.837138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:30.837227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:30.858751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] MFA Rules not processed for user `ff4b7731f96643d0a24e63d31f5f6f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:11:30.860023 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:30.869810 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1329/2677] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:11:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:11:30.916763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:30.918366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:30.976680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f639c679-9329-441b-b447-23dd14dac8ea service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:30.980899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:30.981515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:30.981801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:30.982100 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:30.983887 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:31.000120 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:31.001665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:31.013607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1349/2678] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:31.615153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:31.616548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:31.628707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad94fa-4172-4411-951c-6fb4b24ca7ed None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:31.632205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:31.632801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:31.633075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:31.633904 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:31.635583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:31.651202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:31.652128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:31.664493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1330/2679] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:41.008676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=39 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:41.658286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:11:44.934603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:11:44.944443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50398c9-ccb4-4353-8f70-7f69a9710770 service glance] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:11:44.948108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:11:44.948728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:11:44.949008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:11:44.949282 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:11:44.951068 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:11:44.961415 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:11:44.962925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:11:44.971603 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1350/2680] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:11:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:11:54.968530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:12:12.021248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:12.022979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:12.041662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753a03e-42ca-443b-b96f-8926754463ae service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:12.049069 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:12.049678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:12.049954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:12.050256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:12.052267 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:12.069101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:12.071112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:12.087438 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1331/2681] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:14.223402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:14.237533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18c58e-f469-495d-8fe5-54a2f438ff84 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:14.244486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:14.245244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:14.246305 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:14.246615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:14.249166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:14.266383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:14.269629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:14.286161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1351/2682] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:12:22.082685 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:12:22.821447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:22.830797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e007b-d883-453a-8508-3e7e42c9f8d9 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:22.834471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:22.835214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:22.835554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:22.835888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:22.837662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:22.847214 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:22.848615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:22.856924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1332/2683] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:23.003781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:23.013689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6637730-75a5-4a66-8651-6ce961ae28c4 service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:23.018704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:23.019523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:23.019914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:23.020376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:23.022527 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:23.034604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:23.036307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:23.046146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1352/2684] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:23.131339 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:23.139817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca14e5a0-869b-49d1-8630-254ed3059e99 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:23.143393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:23.143981 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:23.144255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:23.144522 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:23.146252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:23.159503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:23.160912 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:23.172114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1333/2685] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:23.232647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:23.234075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:23.303003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad6ab8c-b629-41c2-9896-15805bf806f7 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:23.307989 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:23.308594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:23.308871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:23.309143 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:23.310868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:23.322024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:23.324110 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:23.333812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1353/2686] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:25.352667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:25.374365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b09eb9e-ffb7-4694-a927-8dbebfa477f7 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:25.379970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:25.380591 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:25.380866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:25.381139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:25.383346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:25.396456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:25.398052 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:25.410097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1334/2687] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:25.976974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:25.986803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3219c4b1-f0db-49ea-8da5-f56c9ad41f1f service placement] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:25.990970 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:25.991566 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:25.991845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:25.992117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:25.993880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:26.006710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:26.008464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:26.019029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1354/2688] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:29.109352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:29.121640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffcdf0-44d4-4cce-a77c-b876d74860e4 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:29.126721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:29.127425 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:29.127767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:29.128104 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:29.129859 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:29.141818 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:29.143424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:29.153473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1335/2689] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:12:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:12:32.411288 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:32.412876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:32.427487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d85d4-d2fe-4679-85c1-9d9f3ff988b6 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:32.433351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:32.434233 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:32.434516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:32.434793 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] PATH_INFO: `/v3/users/6b926f7161c0487a8c2b04ada3b374e3` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:32.435934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b926f7161c0487a8c2b04ada3b374e3)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:32.437872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:32.488277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b926f7161c0487a8c2b04ada3b374e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.496126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b926f7161c0487a8c2b04ada3b374e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.496747 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b926f7161c0487a8c2b04ada3b374e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.497356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b926f7161c0487a8c2b04ada3b374e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.504224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b926f7161c0487a8c2b04ada3b374e3'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.520912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1355/2690] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:12:32 2022] DELETE /identity/v3/users/6b926f7161c0487a8c2b04ada3b374e3 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:12:32.691801 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:32.705235 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734de6-3de9-4c54-9870-7797da52460f service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:32.765706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:32.766293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:32.766576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:32.766853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] PATH_INFO: `/v3/projects/9aff70e0d61c41fbb469ed4bcd785b23` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:32.768012 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] RBAC: Authorizing `identity:delete_project(project_id=9aff70e0d61c41fbb469ed4bcd785b23)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:32.769902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:32.819292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9aff70e0d61c41fbb469ed4bcd785b23', 'request_id': 'req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.824763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9aff70e0d61c41fbb469ed4bcd785b23', 'request_id': 'req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.830881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9aff70e0d61c41fbb469ed4bcd785b23', 'request_id': 'req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.844616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] The token cache is being invalidate because project 9aff70e0d61c41fbb469ed4bcd785b23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:12:32.844903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.845544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.847073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1336/2691] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:12:32 2022] DELETE /identity/v3/projects/9aff70e0d61c41fbb469ed4bcd785b23 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:12:32.851094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:32.852620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:32.866552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769eb2ba-2359-4bdc-8a78-76093f5f8884 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:32.884615 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:12:32.924361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:32.925001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:32.925298 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:32.925556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] PATH_INFO: `/v3/users/b18ba8664afc42e1bdce1f8648f36599` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:32.926719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b18ba8664afc42e1bdce1f8648f36599)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:32.928676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:32.973457 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b18ba8664afc42e1bdce1f8648f36599'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.980946 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b18ba8664afc42e1bdce1f8648f36599'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.981547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b18ba8664afc42e1bdce1f8648f36599'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.982171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b18ba8664afc42e1bdce1f8648f36599'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:32.988919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-dd30e196-2cca-4126-b960-014f924f0f61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b18ba8664afc42e1bdce1f8648f36599'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.005622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1356/2692] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:12:32 2022] DELETE /identity/v3/users/b18ba8664afc42e1bdce1f8648f36599 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:12:33.171903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:12:33.174472 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:12:33.190868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595f87bc-16d8-46e1-b3dd-e9da52fe7ebd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:12:33.224880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:12:33.254873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:12:33.255781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:12:33.258386 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:12:33.258891 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] PATH_INFO: `/v3/projects/0340d1268ba4425396d69ff470e52914` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:12:33.261820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0340d1268ba4425396d69ff470e52914)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:12:33.263879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:12:33.312412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0340d1268ba4425396d69ff470e52914', 'request_id': 'req-7433ea20-ddbb-4945-a028-1540dcd51258'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.318041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0340d1268ba4425396d69ff470e52914', 'request_id': 'req-7433ea20-ddbb-4945-a028-1540dcd51258'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.325178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0340d1268ba4425396d69ff470e52914', 'request_id': 'req-7433ea20-ddbb-4945-a028-1540dcd51258'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.339600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] The token cache is being invalidate because project 0340d1268ba4425396d69ff470e52914 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:12:33.339895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.340524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:12:33.342022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1337/2693] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:12:33 2022] DELETE /identity/v3/projects/0340d1268ba4425396d69ff470e52914 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:12:43.002276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=43,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:12:43.335410 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=34,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:13:10.050307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f75cf-af8f-46f7-8a28-ff1edbcf2212 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:13:10.050817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f75cf-af8f-46f7-8a28-ff1edbcf2212 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:13:10.050928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f75cf-af8f-46f7-8a28-ff1edbcf2212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:13:10.071332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6d8f75cf-af8f-46f7-8a28-ff1edbcf2212 None None] MFA Rules not processed for user `ff4b7731f96643d0a24e63d31f5f6f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:13:10.072564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6d8f75cf-af8f-46f7-8a28-ff1edbcf2212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:10.128315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1357/2694] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:13:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:13:15.115064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:13:15.116830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:15.187830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433ea20-ddbb-4945-a028-1540dcd51258 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:13:15.193841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:13:15.194464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:13:15.194740 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:13:15.195011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:13:15.196665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:15.253098 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:13:15.254723 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:13:15.263639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1338/2695] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:13:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:13:20.121251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=25 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:13:21.946275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:13:21.946880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:13:21.946880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:13:21.965779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] MFA Rules not processed for user `a1f71865b51b4e8480932fffae985567`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:13:21.967079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:21.974224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1358/2696] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:13:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:13:22.277360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:13:22.278809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:22.337152 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a417e5-032e-4ca8-a23a-f49e7d76f722 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:13:22.340963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:13:22.341510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:13:22.341797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:13:22.342099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:13:22.343885 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:13:22.353338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:13:22.354809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:13:22.363251 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1339/2697] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:13:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:13:31.960463 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:13:32.359183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=82 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:18.673077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:18.683258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d102bd55-478c-4603-9112-6fc0d2692e9d None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:18.686781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:18.687375 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:18.687653 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:18.687951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:18.689646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:18.747003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:18.748422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:18.758273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1359/2698] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:14:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:14:23.410009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:23.411406 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:23.421884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b817bf2-cd96-4058-a475-82f3f4dd89e5 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:23.425453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:23.426076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:23.426352 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:23.426623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:23.428236 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:23.490130 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:23.491765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:23.501205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1340/2699] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:14:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:14:28.753405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=28 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:29.649030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:29.650752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:29.664321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a4126-618f-47ae-8e87-f10c2b904473 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:29.668839 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:29.669462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:29.669737 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:29.670031 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] PATH_INFO: `/v3/users/d4ade5932a324a4b8956b5377256ce7a` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:29.671162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4ade5932a324a4b8956b5377256ce7a)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:29.673065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:29.719868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4ade5932a324a4b8956b5377256ce7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:29.727724 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4ade5932a324a4b8956b5377256ce7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:29.728532 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4ade5932a324a4b8956b5377256ce7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:29.729208 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4ade5932a324a4b8956b5377256ce7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:29.736108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4ade5932a324a4b8956b5377256ce7a'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:29.753513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1360/2700] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:14:29 2022] DELETE /identity/v3/users/d4ade5932a324a4b8956b5377256ce7a => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:14:29.922570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:29.932575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb49d8f3-6252-41c7-a18f-089c7df1f9c0 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:29.990324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:29.990964 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:29.991244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:29.991783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] PATH_INFO: `/v3/projects/9edc5ff928624d87944ee071e578aa17` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:29.992905 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9edc5ff928624d87944ee071e578aa17)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:29.994864 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:30.040936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9edc5ff928624d87944ee071e578aa17', 'request_id': 'req-20de1518-1ced-4f95-8bd7-1c235aaa2c83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.046336 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9edc5ff928624d87944ee071e578aa17', 'request_id': 'req-20de1518-1ced-4f95-8bd7-1c235aaa2c83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.052283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9edc5ff928624d87944ee071e578aa17', 'request_id': 'req-20de1518-1ced-4f95-8bd7-1c235aaa2c83'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.063837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] The token cache is being invalidate because project 9edc5ff928624d87944ee071e578aa17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:14:30.064127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.064743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.066243 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1341/2701] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:14:29 2022] DELETE /identity/v3/projects/9edc5ff928624d87944ee071e578aa17 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:14:30.070954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:30.072442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:30.086177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eb4f2a-13df-490f-a34b-32b8d99c7231 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:30.142783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:30.143345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:30.143627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:30.143901 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] PATH_INFO: `/v3/users/9bc019301fea41098084e6d2ce86fc51` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:30.145029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bc019301fea41098084e6d2ce86fc51)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:30.147546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:30.192652 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bc019301fea41098084e6d2ce86fc51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.199983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bc019301fea41098084e6d2ce86fc51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.200587 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bc019301fea41098084e6d2ce86fc51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.201176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bc019301fea41098084e6d2ce86fc51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.208226 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bc019301fea41098084e6d2ce86fc51'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.216818 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:30.228403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1361/2702] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:14:30 2022] DELETE /identity/v3/users/9bc019301fea41098084e6d2ce86fc51 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:14:30.404708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:30.406252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:30.425114 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de1518-1ced-4f95-8bd7-1c235aaa2c83 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:30.447466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:30.487531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:30.488187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:30.488463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:30.488737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] PATH_INFO: `/v3/projects/f998803074924d57b1e6ba81c117cdfd` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:30.489853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f998803074924d57b1e6ba81c117cdfd)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:30.492615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:30.537068 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f998803074924d57b1e6ba81c117cdfd', 'request_id': 'req-b706fc70-ef8b-456b-a955-ea03636eed02'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.542559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f998803074924d57b1e6ba81c117cdfd', 'request_id': 'req-b706fc70-ef8b-456b-a955-ea03636eed02'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.548427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f998803074924d57b1e6ba81c117cdfd', 'request_id': 'req-b706fc70-ef8b-456b-a955-ea03636eed02'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.561412 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] The token cache is being invalidate because project f998803074924d57b1e6ba81c117cdfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:14:30.561698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.562325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-b706fc70-ef8b-456b-a955-ea03636eed02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:14:30.563771 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1342/2703] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:14:30 2022] DELETE /identity/v3/projects/f998803074924d57b1e6ba81c117cdfd => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:14:38.960004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:14:38.961394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:39.028821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63865c2-2b00-41e1-9689-e8d5c0d92ae6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:14:39.087895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:14:39.088515 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:39.088797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:39.089071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:39.090831 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:39.102667 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:14:39.104249 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:14:39.114176 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1362/2704] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:14:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:14:40.558528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:49.110009 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=61,keystone:DELETE=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:14:50.945583 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:14:50.946122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:14:50.946221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:14:50.965419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] MFA Rules not processed for user `ff4b7731f96643d0a24e63d31f5f6f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:14:50.966690 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:14:51.020557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1343/2705] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:14:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:01.014139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=25 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:17.421963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:17.423402 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:17.479170 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb5709-5afa-4c06-ab3d-52010bf723fc service placement] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:17.482852 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:17.483486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:17.483764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:17.484037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:17.485723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:17.546792 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:17.548146 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:17.558080 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1363/2706] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:20.612704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:20.614722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:20.631333 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f681ad-0646-43e0-8470-bf3bda82ea06 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:20.636555 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:20.637192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:20.637511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:20.637766 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] PATH_INFO: `/v3/users/a1f71865b51b4e8480932fffae985567` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:20.638921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1f71865b51b4e8480932fffae985567)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:20.640849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:20.691961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1f71865b51b4e8480932fffae985567'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:20.692621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1f71865b51b4e8480932fffae985567'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:20.702218 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1f71865b51b4e8480932fffae985567'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:20.702935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1f71865b51b4e8480932fffae985567'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:20.711166 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-5c7b1093-16e7-40e2-a2f1-d71cd8e8382f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1f71865b51b4e8480932fffae985567'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:20.730092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1344/2707] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:15:20 2022] DELETE /identity/v3/users/a1f71865b51b4e8480932fffae985567 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:20.906154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:20.916394 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8354b4e-9f1d-4626-b3a3-eae69404a462 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:20.976749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:20.977374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:20.978180 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:20.978451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] PATH_INFO: `/v3/projects/64177183598843a68fc7fbe8257d4881` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:20.979543 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64177183598843a68fc7fbe8257d4881)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:20.981422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.027538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64177183598843a68fc7fbe8257d4881', 'request_id': 'req-2635ba10-c9a6-4e62-98d2-1f5975a32b96'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:21.029366 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:21.034346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64177183598843a68fc7fbe8257d4881', 'request_id': 'req-2635ba10-c9a6-4e62-98d2-1f5975a32b96'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:21.041108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64177183598843a68fc7fbe8257d4881', 'request_id': 'req-2635ba10-c9a6-4e62-98d2-1f5975a32b96'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:21.053086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] The token cache is being invalidate because project 64177183598843a68fc7fbe8257d4881 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:15:21.053423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:21.054081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-2635ba10-c9a6-4e62-98d2-1f5975a32b96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:21.055545 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1364/2708] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:15:20 2022] DELETE /identity/v3/projects/64177183598843a68fc7fbe8257d4881 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:21.061041 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.061357 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.061637 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.080894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:21.082141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:21.141201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1345/2709] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:21.149039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.149323 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.149596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.169097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:21.170303 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:21.177911 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1365/2710] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:21.182886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.192781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a0be9-bb25-4ffa-a9b9-6cab1a35f77c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.197482 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.198118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.198402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.198680 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.199325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:15:21.199636 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.201302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.212688 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1346/2711] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:15:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:15:21.216928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.225878 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cf78fa-b08f-4a64-8ad7-806012055208 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.230945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.231530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.231835 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.232106 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.232760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.234686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.261758 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1366/2712] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:15:21 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:15:21.266016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.276181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf36df-6f93-4395-b75e-f675386ca954 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.330735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.331337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.331612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.331886 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.332511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.334441 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.342836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:21.357666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1347/2713] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:15:21 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:15:21.361794 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.372290 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ea5a29-4c5e-4a83-b3a2-1e0a21c887b5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.376749 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.377342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.377618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.377888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.378548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.380117 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.389790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1367/2714] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:21 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:15:21.393526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.403820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1225dcac-c838-4913-acf3-113cd6495d1f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.408381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.408982 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.409263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.409536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.410210 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.411812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e1fc9-e57f-41ca-9d23-62ab11517001 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.423944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1348/2715] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:15:21.428416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.438319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b1e8d0-5a9c-446a-904f-b53f133611b6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.442801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.443442 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.443717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.443983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] PATH_INFO: `/v3/projects/b49ad4a760764797a6c6cf253facde70/users/6454e73eeb3b4e8a87090a92308f7ecc/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.458241 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b49ad4a760764797a6c6cf253facde70, user_id=6454e73eeb3b4e8a87090a92308f7ecc, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.460856 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.473539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1368/2716] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:15:21 2022] PUT /identity/v3/projects/b49ad4a760764797a6c6cf253facde70/users/6454e73eeb3b4e8a87090a92308f7ecc/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:21.484380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.484657 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.484932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.512630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] MFA Rules not processed for user `6454e73eeb3b4e8a87090a92308f7ecc`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:21.513849 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:21.576502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1349/2717] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:21.587686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:21.589349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:21.661171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407d9b75-4964-473e-be4f-3bd61423ad69 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:21.667279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:21.667877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:21.668157 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:21.668429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:21.676063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:21.677634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79feb48-e103-47ca-873f-674c5574501a service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:21.686616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1369/2718] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:24.347034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:24.406662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9113d044-24fe-49ee-a82e-f6519d37d0cc None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:24.411775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:24.412397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:24.412675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:24.412946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] PATH_INFO: `/v3/users/6454e73eeb3b4e8a87090a92308f7ecc` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:24.414081 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6454e73eeb3b4e8a87090a92308f7ecc)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:24.416638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:24.459798 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6454e73eeb3b4e8a87090a92308f7ecc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:24.460442 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6454e73eeb3b4e8a87090a92308f7ecc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:24.468016 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6454e73eeb3b4e8a87090a92308f7ecc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:24.468722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6454e73eeb3b4e8a87090a92308f7ecc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:24.475872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6454e73eeb3b4e8a87090a92308f7ecc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:24.481536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:24.495980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1350/2719] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:15:24 2022] DELETE /identity/v3/users/6454e73eeb3b4e8a87090a92308f7ecc => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:24.500659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:24.501151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:24.501252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:24.519778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:24.520994 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:24.578660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1370/2720] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:24.588775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:24.590195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:24.646857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709da7d0-a3ef-44b7-bf43-de209bd6eac8 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:24.650987 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:24.651581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:24.651906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:24.652178 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:24.661116 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:24.662575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf65e2-462a-4ff2-8614-da6b2951bd7e service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:24.672683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1351/2721] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:25.183033 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.184482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:25.191454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:25.198569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60eff3b-3fd1-4b75-a0f3-d84218b4564c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.204512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.205346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.205743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.206101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] PATH_INFO: `/v3/projects/b49ad4a760764797a6c6cf253facde70` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.207277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b49ad4a760764797a6c6cf253facde70)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.209251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.255968 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b49ad4a760764797a6c6cf253facde70', 'request_id': 'req-b1929f20-0def-4f56-8304-471422e7eb13'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:25.262043 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b49ad4a760764797a6c6cf253facde70', 'request_id': 'req-b1929f20-0def-4f56-8304-471422e7eb13'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:25.268121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b49ad4a760764797a6c6cf253facde70', 'request_id': 'req-b1929f20-0def-4f56-8304-471422e7eb13'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:25.280284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] The token cache is being invalidate because project b49ad4a760764797a6c6cf253facde70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:15:25.280562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:25.281172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b1929f20-0def-4f56-8304-471422e7eb13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:25.282683 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1371/2722] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:15:25 2022] DELETE /identity/v3/projects/b49ad4a760764797a6c6cf253facde70 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:25.288202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.288487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.288763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.308254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:25.309436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:25.366343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1352/2723] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:25.376078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.376358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.376630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.396359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:25.397481 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:25.405277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1372/2724] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:25.409426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.421365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3efaeb-cfe2-4222-a6bb-aa193b9356ea None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.426118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.426730 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.427007 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.427277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.427922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:15:25.428222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.429922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.442066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1353/2725] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:15:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:15:25.447077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.456330 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56850ed8-4409-4976-8515-a38a4bc7505a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.460827 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.461418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.461694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.462008 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.462627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.464497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.490824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1373/2726] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:15:25 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:15:25.495695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.505484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558854e-048c-478b-9d97-5cd1e0010ffd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.560426 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.561036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.561320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.561597 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.562257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.564369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.585533 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1354/2727] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:15:25 2022] POST /identity/v3/users => generated 293 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:15:25.589625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.599862 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235f05f-d6a7-4d2d-ba83-7dc18c586990 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.604885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.605477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.605754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.606060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.606693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.608300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.619884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1374/2728] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:15:25.622736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.632776 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b69f3a-e95a-4aa2-9a49-b75633d0b2c6 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.637379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.638001 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.638275 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.638545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.639181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.640759 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0af414-0a29-48ad-88d3-255d075a0a99 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.641994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:25.653297 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1355/2729] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:15:25.657006 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.667552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8054f9-dcbf-41eb-b7eb-23371203b170 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.672251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.672904 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.673182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.673453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] PATH_INFO: `/v3/projects/b889f06355664d5cba8eac5271d0c552/users/7e9ff6f89c604ef58f75445efc587cfc/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.688277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] RBAC: Authorizing `identity:create_grant(project_id=b889f06355664d5cba8eac5271d0c552, user_id=7e9ff6f89c604ef58f75445efc587cfc, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.690942 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.703548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1375/2730] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:15:25 2022] PUT /identity/v3/projects/b889f06355664d5cba8eac5271d0c552/users/7e9ff6f89c604ef58f75445efc587cfc/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:25.714107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.714393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.714666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.742100 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] MFA Rules not processed for user `7e9ff6f89c604ef58f75445efc587cfc`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:25.744237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:25.805222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1356/2731] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:25.816821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:25.818540 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:25.895238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0989cd-17c4-41e0-8fca-f49fbf72c6df admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:25.898844 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:25.899952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:25.900231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:25.900500 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:25.908645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:25.910022 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719909e-7327-46e6-8895-da63f5c49a28 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:25.919220 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1376/2732] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:25 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:28.457890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:28.519923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1353a68d-5a20-45a8-bb52-716483726cc8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:28.524384 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:28.525005 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:28.525833 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:28.526134 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] PATH_INFO: `/v3/users/7e9ff6f89c604ef58f75445efc587cfc` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:28.527250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e9ff6f89c604ef58f75445efc587cfc)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:28.529201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:28.574233 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e9ff6f89c604ef58f75445efc587cfc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:28.574894 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e9ff6f89c604ef58f75445efc587cfc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:28.581579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e9ff6f89c604ef58f75445efc587cfc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:28.582281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e9ff6f89c604ef58f75445efc587cfc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:28.589792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e9ff6f89c604ef58f75445efc587cfc'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:28.606711 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1357/2733] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:15:28 2022] DELETE /identity/v3/users/7e9ff6f89c604ef58f75445efc587cfc => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:28.611338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:28.611797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:28.611910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:28.620063 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:28.632608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:28.633896 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:28.689866 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1377/2734] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:28.699958 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:28.701349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:28.707847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:28.759924 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68eda26-1809-4d95-ac9e-6ce5232daad0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:28.764208 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:28.764851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:28.765129 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:28.765402 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:28.775720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:28.777732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec805e03-7edf-4216-8b78-dffbcaf36acc service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:28.787697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1358/2735] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:29.046041 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.047622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:29.059433 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a75cf-1551-4f0c-98fa-9a481dbbf11a None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.064113 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.064826 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.065114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.065387 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] PATH_INFO: `/v3/projects/b889f06355664d5cba8eac5271d0c552` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.066562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=b889f06355664d5cba8eac5271d0c552)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.068466 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.114948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b889f06355664d5cba8eac5271d0c552', 'request_id': 'req-9e2d9348-2185-4c45-8371-966062b5c0ba'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:29.120681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b889f06355664d5cba8eac5271d0c552', 'request_id': 'req-9e2d9348-2185-4c45-8371-966062b5c0ba'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:29.126761 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b889f06355664d5cba8eac5271d0c552', 'request_id': 'req-9e2d9348-2185-4c45-8371-966062b5c0ba'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:29.139005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] The token cache is being invalidate because project b889f06355664d5cba8eac5271d0c552 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:15:29.139292 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:29.139939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9e2d9348-2185-4c45-8371-966062b5c0ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:15:29.141417 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1378/2736] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:15:29 2022] DELETE /identity/v3/projects/b889f06355664d5cba8eac5271d0c552 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:29.146261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.146552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.146830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.166407 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:29.168190 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:29.225202 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1359/2737] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:29.233027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.233315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.233591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.253276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:29.254499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:29.262898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1379/2738] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:29.267158 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.277683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6309575-f459-49e6-b7b6-7e5efc4bb09d None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.282485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.283090 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.283366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.283633 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.284274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:15:29.284577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.286245 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.297506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1360/2739] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:15:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:15:29.301759 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.311001 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22fa40b-94f2-424c-8e35-da9f30f7e2e0 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.315598 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.316194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.316473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.316745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.317383 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.319304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.346651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1380/2740] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:15:29 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:15:29.350764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.360434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ddc6a-bb0f-4361-8c6a-ab091b930a77 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.415117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.415644 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.415920 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.416187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.416839 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.418790 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.439575 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1361/2741] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:15:29 2022] POST /identity/v3/users => generated 285 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:15:29.443783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.454533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6bc1c-8c9d-4985-a0c4-ac343ddb8bd6 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.459073 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.459670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.459949 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.460222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.460859 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.462484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.472740 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1381/2742] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:15:29.476476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.486417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2415b888-0861-4274-9aa4-cd3b8a67a9d7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.490979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.491565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.491841 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.492110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.493004 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.494672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de0baa7-586e-4a71-b1c3-037b374776b3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.504871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1362/2743] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:15:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:15:29.508808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.519012 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26178021-dabb-40dd-8fc5-046ee17e982c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.523536 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.524175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.524455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.524728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] PATH_INFO: `/v3/projects/b6e1ae312d644c5ba6dbebc93838b59f/users/230e498f82c74c879d6f4e5f5c8cfe4b/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.539782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6e1ae312d644c5ba6dbebc93838b59f, user_id=230e498f82c74c879d6f4e5f5c8cfe4b, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.542461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.555264 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1382/2744] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:15:29 2022] PUT /identity/v3/projects/b6e1ae312d644c5ba6dbebc93838b59f/users/230e498f82c74c879d6f4e5f5c8cfe4b/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:15:29.566998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.567421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.567698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.592428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:29.595775 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] MFA Rules not processed for user `230e498f82c74c879d6f4e5f5c8cfe4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:15:29.597599 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:29.660268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1363/2745] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:15:29 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:15:29.671782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:15:29.673461 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:15:29.700851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:29.754003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e869d-d0e1-4fa1-90d2-fa2a3bb4b3f1 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:15:29.757944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:15:29.758687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:15:29.759011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:15:29.759287 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:15:29.769038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:15:29.770821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661f17f-009e-4cfc-ac51-7d131073569c service glance] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:15:29.780069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1383/2746] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:15:29 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:15:39.657411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=26 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:15:39.775564 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=24 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:02.758892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:02.819436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09ddf06-d54f-4ff1-a794-e14d51ca3dc4 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:02.823869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:02.824485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:02.824758 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:02.825026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] PATH_INFO: `/v3/users/230e498f82c74c879d6f4e5f5c8cfe4b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:02.826148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] RBAC: Authorizing `identity:delete_user(user_id=230e498f82c74c879d6f4e5f5c8cfe4b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:02.828205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:02.874085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '230e498f82c74c879d6f4e5f5c8cfe4b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:02.874737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '230e498f82c74c879d6f4e5f5c8cfe4b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:02.881604 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '230e498f82c74c879d6f4e5f5c8cfe4b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:02.882264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '230e498f82c74c879d6f4e5f5c8cfe4b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:02.889040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '230e498f82c74c879d6f4e5f5c8cfe4b'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:02.905762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1364/2747] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:16:02 2022] DELETE /identity/v3/users/230e498f82c74c879d6f4e5f5c8cfe4b => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:16:02.910276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:02.910763 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:02.910875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:02.929829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:02.931037 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:02.987811 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1384/2748] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:02.997732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:02.999749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.055212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6545a-2722-48d2-b162-b9843d4d450c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.058844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.059443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.059721 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.059996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.069421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.070876 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3d1b5b-b118-49a0-a6c9-f3fa1d774002 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.080705 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1365/2749] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:16:03.347160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.348787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.363924 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bee9a29-7c91-4f60-b21b-05ace7d1d1ce None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.369165 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.370936 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.371221 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.372114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] PATH_INFO: `/v3/projects/b6e1ae312d644c5ba6dbebc93838b59f` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.372704 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6e1ae312d644c5ba6dbebc93838b59f)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.375150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.425134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6e1ae312d644c5ba6dbebc93838b59f', 'request_id': 'req-29ae83e7-df59-463e-b195-73f7df97d3ad'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:03.430626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6e1ae312d644c5ba6dbebc93838b59f', 'request_id': 'req-29ae83e7-df59-463e-b195-73f7df97d3ad'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:03.437131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6e1ae312d644c5ba6dbebc93838b59f', 'request_id': 'req-29ae83e7-df59-463e-b195-73f7df97d3ad'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:03.449223 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] The token cache is being invalidate because project b6e1ae312d644c5ba6dbebc93838b59f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:16:03.449512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:03.450193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-29ae83e7-df59-463e-b195-73f7df97d3ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:16:03.452354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1385/2750] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:16:03 2022] DELETE /identity/v3/projects/b6e1ae312d644c5ba6dbebc93838b59f => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:16:03.457595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.457882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.458189 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.478428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:03.479617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.496054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:03.539424 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1366/2751] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:03.547355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.547639 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.547914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:03.568845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.576608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1386/2752] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:03.580827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.590344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6394a9-fa51-4035-bfa9-674bbd2e3811 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.594937 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.595532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.595813 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.596087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.596737 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:16:03.597040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.598672 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.610375 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1367/2753] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:16:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:16:03.614498 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.625144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bc725b-c452-437c-8529-6a9a5bb5d2eb None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.631959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.632691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.633066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.633422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.634164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.636132 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.663069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1387/2754] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:16:03 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:16:03.667259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.677665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f850cbd-c32a-4c98-8144-91f6851ffe7f admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.732422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.733577 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.733854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.734155 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.734797 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.736702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.757999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1368/2755] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:16:03 2022] POST /identity/v3/users => generated 291 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:16:03.762062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.772518 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178c589-20f1-48fe-b591-08bc89f42dae admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.777066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.777658 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.777935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.778236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.778873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.780454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.790269 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1388/2756] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:16:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:16:03.794126 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.804693 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5d39e-e0e6-460d-bcfd-8468e2f4a2aa admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.809378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.810011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.810293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.810570 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.811830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.813428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b81bd9-be55-40b8-a8cc-d3372db4e2cf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.824265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1369/2757] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:16:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:16:03.828188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.837965 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9c130d-a4a2-4543-8bdf-01287919eb26 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:03.842592 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:03.843228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.843513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.843785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] PATH_INFO: `/v3/projects/33af503fc7e9410885f704ffbfcb900b/users/c689e56cfc3f43e187c8804fd8577cb7/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.859512 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33af503fc7e9410885f704ffbfcb900b, user_id=c689e56cfc3f43e187c8804fd8577cb7, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:03.862147 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:03.876448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1389/2758] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:16:03 2022] PUT /identity/v3/projects/33af503fc7e9410885f704ffbfcb900b/users/c689e56cfc3f43e187c8804fd8577cb7/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:16:03.881355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510a2fa-6cad-4292-9007-39d8612af47b None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:03.881641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510a2fa-6cad-4292-9007-39d8612af47b None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:03.881919 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510a2fa-6cad-4292-9007-39d8612af47b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:03.900715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-7510a2fa-6cad-4292-9007-39d8612af47b None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:03.901926 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7510a2fa-6cad-4292-9007-39d8612af47b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.959440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1370/2759] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:03.969581 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:03.971272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:03.993299 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:04.028928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963959b4-a785-4960-8a56-49630f7b09d5 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:04.033027 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:04.033625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:04.033900 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:04.034195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:04.043024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:04.044376 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:04.054200 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1390/2760] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:16:04.705944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e060a-d27a-4808-b776-81ae77c6f98f None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:04.706266 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e060a-d27a-4808-b776-81ae77c6f98f None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:04.706547 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e060a-d27a-4808-b776-81ae77c6f98f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:04.734811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b05e060a-d27a-4808-b776-81ae77c6f98f None None] MFA Rules not processed for user `c689e56cfc3f43e187c8804fd8577cb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:04.736046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b05e060a-d27a-4808-b776-81ae77c6f98f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:04.753950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:04.803697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1371/2761] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:04.815028 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:04.816426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:04.874845 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a144f98-778d-4568-9557-9f50e8b3e2ca service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:04.878334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:04.878937 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:04.879215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:04.879488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:04.887672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:04.889710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:04.898435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1391/2762] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:16:06.250034 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af3ac6-6b2e-4071-93b7-18fffcff1e46 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:06.250567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af3ac6-6b2e-4071-93b7-18fffcff1e46 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:06.250567 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31af3ac6-6b2e-4071-93b7-18fffcff1e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:06.269278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-31af3ac6-6b2e-4071-93b7-18fffcff1e46 None None] MFA Rules not processed for user `c689e56cfc3f43e187c8804fd8577cb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:06.270504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-31af3ac6-6b2e-4071-93b7-18fffcff1e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:06.277246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1372/2763] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:14.894577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=43 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:16.263811 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=22 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:28.076801 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:28.078252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:28.159506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb1530-0865-4dee-9102-e101cb4091ad service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:28.165056 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:28.165672 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:28.165950 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:28.166251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:28.167987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:28.225870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:28.227473 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:28.236427 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1392/2764] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:16:33.567309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:33.567847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:33.567847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:33.587070 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] MFA Rules not processed for user `ff4b7731f96643d0a24e63d31f5f6f00`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:16:33.588252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:33.595091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1373/2765] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:16:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:16:33.632939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:33.634453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:33.691907 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8c96d7-4a68-43e8-b636-2c7e036d5c94 service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:33.695468 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '589508bdaf3c4e159c01e3a43eb69392', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:33.696069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:33.696344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:33.696614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:33.698342 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:33.711943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:33.713355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:33.724728 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1393/2766] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:16:35.323307 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:35.339518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72338f5f-b9af-418c-9699-2ea14fbf759c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:35.343566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:35.344274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:35.344559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:35.344848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:35.347478 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:35.363099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:35.364524 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:35.377922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1374/2767] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:16:43.719353 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=82 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:45.266214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:16:45.267584 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:45.279541 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b61ad-e705-4617-929d-701964aaec53 service glance] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:16:45.283109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:16:45.283711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:16:45.284018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:16:45.284291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:16:45.286016 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:16:45.295326 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:16:45.296730 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0978360-626f-447b-8a64-79172cbaae8e service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:16:45.304636 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1394/2768] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:16:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:16:45.371054 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=9 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:16:55.301053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:17:13.255566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:17:13.257857 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:13.272465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69833e-a636-4b73-be33-0b3cf3b790e8 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:17:13.279161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:17:13.280483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:17:13.280773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:17:13.281046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:17:13.283138 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:13.298881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:17:13.298881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:17:13.310197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1375/2769] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:17:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:17:23.306504 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:17:24.125368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:17:24.125884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:17:24.125884 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:17:24.144779 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] MFA Rules not processed for user `c689e56cfc3f43e187c8804fd8577cb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:17:24.146082 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:24.152838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1395/2770] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:17:24 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:17:24.683459 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:17:24.684955 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:24.755835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb64ef-cb03-487e-ad61-bcb730c9a019 service nova] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:17:24.760744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:17:24.761343 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:17:24.761619 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:17:24.761890 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:17:24.763588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:24.775133 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:17:24.776694 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:17:24.786892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1376/2771] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:17:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:17:33.353088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:17:33.367401 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742838ac-d26f-4893-a996-2d4da8845264 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:17:33.373204 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:17:33.373815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:17:33.374126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:17:33.374400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:17:33.376127 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:17:33.389172 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:17:33.390917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:17:33.402380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1396/2772] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:17:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:17:34.782175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=32 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:17:43.397450 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=9 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:27.544526 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:27.545935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:27.558525 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521f4918-5686-4717-ad30-dc14ef3728cf service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:27.563545 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:27.564132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:27.564413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:27.564689 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:27.566616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:27.578998 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:27.580747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:27.592422 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1377/2773] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:27 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:18:27.595266 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:27.607501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2723ac1c-c367-45f5-8901-3d4afe592dea service nova] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:27.611668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:27.612318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:27.612872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:27.612872 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] PATH_INFO: `/v3/users/c689e56cfc3f43e187c8804fd8577cb7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:27.614538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c689e56cfc3f43e187c8804fd8577cb7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:27.616659 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:27.662467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c689e56cfc3f43e187c8804fd8577cb7'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.669232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c689e56cfc3f43e187c8804fd8577cb7'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.669850 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c689e56cfc3f43e187c8804fd8577cb7'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.670486 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c689e56cfc3f43e187c8804fd8577cb7'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.677456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-caafbb29-f216-423e-8818-d8c36525d790 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c689e56cfc3f43e187c8804fd8577cb7'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.694409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1397/2774] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:18:27 2022] DELETE /identity/v3/users/c689e56cfc3f43e187c8804fd8577cb7 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:27.863389 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:27.876290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a458a8-90b8-4bc8-86f2-13fa8323d3e5 service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:27.933963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:27.934565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:27.934848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:27.935124 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] PATH_INFO: `/v3/projects/33af503fc7e9410885f704ffbfcb900b` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:27.936263 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33af503fc7e9410885f704ffbfcb900b)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:27.938205 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:27.990392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33af503fc7e9410885f704ffbfcb900b', 'request_id': 'req-a126ed11-6a0e-4411-a723-c0fe6d375a93'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:27.996042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33af503fc7e9410885f704ffbfcb900b', 'request_id': 'req-a126ed11-6a0e-4411-a723-c0fe6d375a93'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:28.001994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33af503fc7e9410885f704ffbfcb900b', 'request_id': 'req-a126ed11-6a0e-4411-a723-c0fe6d375a93'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:28.014318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] The token cache is being invalidate because project 33af503fc7e9410885f704ffbfcb900b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:18:28.014613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:28.015248 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-a126ed11-6a0e-4411-a723-c0fe6d375a93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:28.016710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1378/2775] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:18:27 2022] DELETE /identity/v3/projects/33af503fc7e9410885f704ffbfcb900b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:28.022313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.022607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.022886 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.041948 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:28.043168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:28.100159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1398/2776] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:28.108944 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.109234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.109508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.130953 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:28.132164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:28.140049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1379/2777] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:28.144453 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.154410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188f95a5-b669-42f7-9050-4e6e2acb92f8 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.159088 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.159720 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.160000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.160276 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.160920 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:18:28.161338 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.162988 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.174604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1399/2778] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:18:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:18:28.179488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.188781 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b2ba6-4655-46bd-9d0a-0c826971db22 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.193943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.194587 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.194873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.195169 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.195803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.197683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.224928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1380/2779] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:18:28 2022] POST /identity/v3/projects => generated 391 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:18:28.228983 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.238880 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864daea-5b31-414f-9bc2-7656a472dfe0 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.294993 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.295604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.295885 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.296164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.296783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.298722 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.307069 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:28.324910 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1400/2780] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:18:28 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:18:28.326113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.336661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea05f0f-2c40-43ab-93eb-daf86b796b2c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.341139 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.341736 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.342042 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.342312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.344528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.354132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1381/2781] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:18:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:18:28.357899 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.368150 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b9146-ae41-4684-95e9-bf100712aa21 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.372909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.373497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.373774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.374072 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.374702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.376273 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28403454-7225-46b6-85ad-0bd91670cd84 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.387451 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1401/2782] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:18:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:18:28.391449 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.401128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00732a70-77f7-4dac-8d55-89fa7782c3cf admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.405676 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.406344 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.406621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.406888 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] PATH_INFO: `/v3/projects/b2aa9d7a38db4783b343ac1467a7bf47/users/df18c05056244654a3357b680f56fcf4/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.420935 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2aa9d7a38db4783b343ac1467a7bf47, user_id=df18c05056244654a3357b680f56fcf4, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.423586 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.437212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1382/2783] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:18:28 2022] PUT /identity/v3/projects/b2aa9d7a38db4783b343ac1467a7bf47/users/df18c05056244654a3357b680f56fcf4/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:28.447829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.448054 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.448341 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.476670 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] MFA Rules not processed for user `df18c05056244654a3357b680f56fcf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:28.477865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:28.539975 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1402/2784] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:28 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:18:28.550595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:28.552979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:28.567274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:28.611085 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74563b7-7b15-4f91-a61c-c95ad45eac69 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:28.615291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:28.615913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:28.616191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:28.616462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:28.624910 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:28.626371 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031ff7bb-ad92-444a-9f1d-81e1f3364c18 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:28.635057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1383/2785] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:28 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:18:38.536400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:INSERT=3,keystone:SELECT=39 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:38.631060 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=20 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:49.946002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:50.009614 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbd1ccf-d4b7-4a00-a63b-20f4207dc761 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:50.014077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:50.014706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:50.014999 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:50.015254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] PATH_INFO: `/v3/users/df18c05056244654a3357b680f56fcf4` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:50.016365 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=df18c05056244654a3357b680f56fcf4)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:50.018313 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:50.067696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df18c05056244654a3357b680f56fcf4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.077187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df18c05056244654a3357b680f56fcf4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.078021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df18c05056244654a3357b680f56fcf4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.078611 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df18c05056244654a3357b680f56fcf4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.087526 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df18c05056244654a3357b680f56fcf4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.106791 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1403/2786] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:18:49 2022] DELETE /identity/v3/users/df18c05056244654a3357b680f56fcf4 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:50.111252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:50.112187 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:50.112702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:50.133751 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:50.134934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:50.201596 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1384/2787] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:50.211797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:50.213239 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:50.280369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085cde02-77a4-4c31-a31b-b94156a399eb admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:50.284183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:50.284797 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:50.285130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:50.285409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:50.294746 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:50.296169 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:50.305842 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1404/2788] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:18:50.586423 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:50.587963 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:50.602089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91e15c-98f0-4b8f-9bcb-6d22dcefa4eb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:50.606902 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:50.607695 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:50.607976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:50.608836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] PATH_INFO: `/v3/projects/b2aa9d7a38db4783b343ac1467a7bf47` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:50.610866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2aa9d7a38db4783b343ac1467a7bf47)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:50.612855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:50.667403 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2aa9d7a38db4783b343ac1467a7bf47', 'request_id': 'req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.675257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2aa9d7a38db4783b343ac1467a7bf47', 'request_id': 'req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.681066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2aa9d7a38db4783b343ac1467a7bf47', 'request_id': 'req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.697301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] The token cache is being invalidate because project b2aa9d7a38db4783b343ac1467a7bf47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:18:50.697678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.698995 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:50.703989 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1385/2789] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:18:50 2022] DELETE /identity/v3/projects/b2aa9d7a38db4783b343ac1467a7bf47 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:54.982368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:54.991005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13027912-53e5-42ae-9d87-6be50d5db93d service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:55.009524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:55.041380 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:55.042011 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:55.042297 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:55.042570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:55.044277 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:55.104161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:55.105573 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:55.115000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1405/2790] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:18:58.105113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.107918 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:58.179482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.181549 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:58.195293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6034dc0-6752-49fd-80e8-bfd8fa5a37ba admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.198173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f3568f-1f25-4755-b191-36226ecb52a8 service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.200961 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.201574 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.201856 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.202224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.203337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.203923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:58.204039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.204228 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.204497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] PATH_INFO: `/v3/users/ff4b7731f96643d0a24e63d31f5f6f00` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.205596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff4b7731f96643d0a24e63d31f5f6f00)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.207690 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.217381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.219148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.229506 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1386/2791] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:18:58.266602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff4b7731f96643d0a24e63d31f5f6f00'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.275328 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff4b7731f96643d0a24e63d31f5f6f00'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.276148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff4b7731f96643d0a24e63d31f5f6f00'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.276815 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff4b7731f96643d0a24e63d31f5f6f00'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.284301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-f6551451-cca8-4de8-9a84-72aa2656a747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff4b7731f96643d0a24e63d31f5f6f00'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.302663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1406/2792] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:18:58 2022] DELETE /identity/v3/users/ff4b7731f96643d0a24e63d31f5f6f00 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:18:58.482611 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.498934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbf1e79-5b15-4e78-9350-be851077b08c service placement] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.509655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:58.562466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.563198 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.563566 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.563747 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] PATH_INFO: `/v3/projects/68345feb30fe454083122ecaf73ef747` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.564869 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] RBAC: Authorizing `identity:delete_project(project_id=68345feb30fe454083122ecaf73ef747)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.566820 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.614715 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68345feb30fe454083122ecaf73ef747', 'request_id': 'req-d4ad79c9-890e-4610-8da7-38fc3f78a434'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.620977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68345feb30fe454083122ecaf73ef747', 'request_id': 'req-d4ad79c9-890e-4610-8da7-38fc3f78a434'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.627149 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68345feb30fe454083122ecaf73ef747', 'request_id': 'req-d4ad79c9-890e-4610-8da7-38fc3f78a434'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.639107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] The token cache is being invalidate because project 68345feb30fe454083122ecaf73ef747 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:18:58.639396 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.640011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-d4ad79c9-890e-4610-8da7-38fc3f78a434 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:58.641483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1387/2793] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:18:58 2022] DELETE /identity/v3/projects/68345feb30fe454083122ecaf73ef747 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:58.646449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.646733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.647005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.667238 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:58.668423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:58.707181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:58.727621 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1407/2794] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:58.735946 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.736232 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.736507 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.757063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:58.758349 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:58.766095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1388/2795] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:58.770236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.781355 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5de8461-393c-44ff-a26a-2334391e4720 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.785876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.786632 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.786912 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.787185 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.787830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:18:58.788847 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.790930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.804316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1408/2796] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:18:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:18:58.808594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.817828 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d457f2-51c3-4a44-a518-ab323d5d2302 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.822614 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.823216 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.823493 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.823764 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.824443 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.826348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.855197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1389/2797] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:18:58 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:18:58.859389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.869351 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb12e62-20c5-4ba5-b805-19549016842d admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.924203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.924798 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.925075 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.925344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.925982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.927894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.949109 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1409/2798] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:18:58 2022] POST /identity/v3/users => generated 288 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:18:58.953286 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.963976 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead5053-3223-4f78-b6c6-44b116d60536 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:58.968411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:58.969009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:58.969283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:58.969552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:58.970212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:58.971779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:58.982368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1390/2799] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:18:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:18:58.985953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:58.996024 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213c5ab-d50e-431b-b882-45667f8237a3 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:59.000583 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:59.001173 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.001460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.001734 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.002409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:59.003978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357082cf-824e-4ad9-8400-3bf94932b5a2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:59.014717 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1410/2800] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:18:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:18:59.018531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:59.028763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5d712f-659a-4046-8b3a-554ca9699142 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:59.033300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:59.033934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.034239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.034515 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] PATH_INFO: `/v3/projects/72e52c52a2524fc4a69040c818109867/users/c25289bd54b24b7dabb300a53ccb4959/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.169930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=72e52c52a2524fc4a69040c818109867, user_id=c25289bd54b24b7dabb300a53ccb4959, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:59.172576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:59.185262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1391/2801] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:18:59 2022] PUT /identity/v3/projects/72e52c52a2524fc4a69040c818109867/users/c25289bd54b24b7dabb300a53ccb4959/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:59.196257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.196742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.196846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.225195 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] MFA Rules not processed for user `c25289bd54b24b7dabb300a53ccb4959`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:59.226356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:59.289995 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1411/2802] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:59.301365 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:59.303047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:59.345569 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:59.375319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f7dc56-a430-47d7-a18f-24d14ea425ca admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:59.380696 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6adb6b4eff2545c080d2eb3327b78d0f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'ResellerAdmin', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:59.381290 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.381563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.382392 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.389871 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:59.391453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051c9623-8bfb-4009-85b1-d4bf16cedfd1 service nova] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:59.400488 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1392/2803] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:18:59.610917 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:59.654160 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:18:59.674174 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768b0221-e63e-4ff9-afcc-447377e3aaaa None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:59.678637 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:59.679774 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.680050 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.680319 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] PATH_INFO: `/v3/users/c25289bd54b24b7dabb300a53ccb4959` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.681423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c25289bd54b24b7dabb300a53ccb4959)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:59.683373 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:59.727727 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c25289bd54b24b7dabb300a53ccb4959'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:59.735164 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c25289bd54b24b7dabb300a53ccb4959'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:59.735796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c25289bd54b24b7dabb300a53ccb4959'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:59.736409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c25289bd54b24b7dabb300a53ccb4959'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:59.743775 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c25289bd54b24b7dabb300a53ccb4959'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:18:59.762065 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1412/2804] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:18:59 2022] DELETE /identity/v3/users/c25289bd54b24b7dabb300a53ccb4959 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:18:59.766461 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.766749 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.767020 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.785277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:18:59.786528 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:59.843617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1393/2805] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:18:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:18:59.853324 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:18:59.854777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:18:59.912991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17b394f-f5e4-4a9d-8460-6399127fef13 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:18:59.917275 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:18:59.917877 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:18:59.918219 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:18:59.918493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:18:59.927397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:18:59.928858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be48be86-512f-467c-93a7-e4b391daf08c service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:18:59.938855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1413/2806] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:18:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:00.214773 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.216262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:00.228805 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d18b18-6335-477b-91b8-13db55aeba1a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.233411 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.234180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.234463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.234738 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] PATH_INFO: `/v3/projects/72e52c52a2524fc4a69040c818109867` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.236180 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72e52c52a2524fc4a69040c818109867)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.238148 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.287066 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72e52c52a2524fc4a69040c818109867', 'request_id': 'req-ae8381b6-58d4-4678-ae6a-21b43ecb5263'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:00.293107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72e52c52a2524fc4a69040c818109867', 'request_id': 'req-ae8381b6-58d4-4678-ae6a-21b43ecb5263'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:00.299157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72e52c52a2524fc4a69040c818109867', 'request_id': 'req-ae8381b6-58d4-4678-ae6a-21b43ecb5263'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:00.310979 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] The token cache is being invalidate because project 72e52c52a2524fc4a69040c818109867 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:00.311265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:00.311882 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ae8381b6-58d4-4678-ae6a-21b43ecb5263 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:00.313337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1394/2807] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:00 2022] DELETE /identity/v3/projects/72e52c52a2524fc4a69040c818109867 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:00.318094 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.318616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.318616 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.337898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:00.339108 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:00.396428 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1414/2808] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:00.404704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.405058 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.405362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.425159 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:00.426397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:00.434326 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1395/2809] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:00.438232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.442914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:00.449199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d275142-81ed-4a70-be05-a60511d55d57 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.454925 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.455525 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.455803 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.456074 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.456710 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:00.457010 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.458651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.471107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1415/2810] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:19:00.475296 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.484836 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1411a-f969-4712-95a5-0c2f613017f1 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.489393 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.489999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.490293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.490563 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.491196 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.493073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.519256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1396/2811] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/projects => generated 363 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:00.524131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.534449 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987c0fc7-6b66-41b6-927e-c7bf9bd819a9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.591429 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.592078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.592372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.592650 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.593271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.595224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.616934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1416/2812] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/users => generated 282 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:19:00.621417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.631550 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6f8d94-46ce-4cf2-be7c-cfaba9168f9e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.636568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.637164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.637439 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.637708 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.638381 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.639954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.650234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1397/2813] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:00.653966 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.664116 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6968ae-45b6-4748-bfd4-63f9462ca291 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.668754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.669350 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.669633 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.669908 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.670607 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.672187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c72d4eb-2813-40ba-ab2d-76a87f5d9ddb admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.682686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1417/2814] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:00.686779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.696854 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80d5c27-1971-4b7a-8a4e-4553e205673e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.701602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.702281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.702562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.702835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] PATH_INFO: `/v3/projects/fab6800c0a424d719895d1277603a69a/users/beb753fca7d84f7c83c340c7692120d0/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.706495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:00.719338 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fab6800c0a424d719895d1277603a69a, user_id=beb753fca7d84f7c83c340c7692120d0, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.722183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.734728 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1398/2815] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:00 2022] PUT /identity/v3/projects/fab6800c0a424d719895d1277603a69a/users/beb753fca7d84f7c83c340c7692120d0/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:00.746422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.746708 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.746982 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.775372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] MFA Rules not processed for user `beb753fca7d84f7c83c340c7692120d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:00.776591 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:00.840310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1418/2816] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:00.844947 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.907197 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ed9e3a-d6ad-417d-b49d-b316d3c15952 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:00.911768 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:00.912387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:00.912646 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:00.912970 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:00.913698 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:00.915809 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:00.942039 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1399/2817] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/projects => generated 365 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:00.946626 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:00.957112 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40d600-c98f-4631-822b-58294fa199dc None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.005834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:01.012853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.013456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.014102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.014572 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.015393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.017291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.040346 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1419/2818] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:00 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:01.042595 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.052720 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29737bd-ffbd-4db2-9c3e-dd7ab4fd784e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.057883 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.058502 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.058779 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.059047 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.059684 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.061258 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.071487 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1400/2819] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:19:01.075410 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.085144 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3640ea6-2121-4695-a6fd-11e3d2f72d9f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.089711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.090400 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.090681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.090954 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] PATH_INFO: `/v3/projects/bc686f35b7954a7ca7047211804e2510/users/901bd88097f24e15b4a38f84c26df2dd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.106405 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc686f35b7954a7ca7047211804e2510, user_id=901bd88097f24e15b4a38f84c26df2dd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.109053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.122928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1420/2820] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:01 2022] PUT /identity/v3/projects/bc686f35b7954a7ca7047211804e2510/users/901bd88097f24e15b4a38f84c26df2dd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:01.127009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.137479 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97743c15-510f-4b51-a48f-87b1b71453eb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.193548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.194182 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.194520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.195291 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.195922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.197510 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a505f5e-1602-4b13-b9de-f70a0c6add4b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.206913 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1401/2821] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:01.211005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.220976 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133ff9d1-cd1a-4839-b4ab-cae56364d385 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.225520 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.226224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.226503 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.226773 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] PATH_INFO: `/v3/projects/bc686f35b7954a7ca7047211804e2510/users/901bd88097f24e15b4a38f84c26df2dd/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.228663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc686f35b7954a7ca7047211804e2510, user_id=901bd88097f24e15b4a38f84c26df2dd, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.231304 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.241781 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1421/2822] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:01 2022] PUT /identity/v3/projects/bc686f35b7954a7ca7047211804e2510/users/901bd88097f24e15b4a38f84c26df2dd/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:01.252226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.252732 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.252838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.281576 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] MFA Rules not processed for user `901bd88097f24e15b4a38f84c26df2dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:01.282835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:01.325231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:01.349026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1402/2823] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:01 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:01.360423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.362166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:01.419280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2e91c8-cd02-4226-863c-52bc2826674a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.423040 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.423634 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.423909 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.424182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.433456 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.434895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c48f4-6b19-477d-85bf-668ea3787c57 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.444224 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1422/2824] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:01 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:01.845387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:01.907948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e307f0-5613-46e4-81d2-78552df9878a None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:01.912399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:01.913032 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.913310 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.913585 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] PATH_INFO: `/v3/users/beb753fca7d84f7c83c340c7692120d0` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:01.915324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=beb753fca7d84f7c83c340c7692120d0)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:01.917262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:01.961714 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'beb753fca7d84f7c83c340c7692120d0'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:01.962436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'beb753fca7d84f7c83c340c7692120d0'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:01.969877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'beb753fca7d84f7c83c340c7692120d0'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:01.970520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'beb753fca7d84f7c83c340c7692120d0'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:01.977951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'beb753fca7d84f7c83c340c7692120d0'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:01.994458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1403/2825] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:01 2022] DELETE /identity/v3/users/beb753fca7d84f7c83c340c7692120d0 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:01.999258 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:01.999546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:01.999825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:02.018196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:02.019397 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:02.080980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1423/2826] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:02.091120 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:02.092463 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:02.150980 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff3b403-1230-418e-b023-68ef0d929ed0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:02.155053 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:02.155675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:02.155934 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:02.156252 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:02.165325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:02.166807 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:02.175941 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1404/2827] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:02.445467 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:02.446953 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:02.461262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37604c7b-6fa4-4ac9-9ae8-901112f32750 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:02.466516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:02.467171 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:02.467974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:02.468254 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] PATH_INFO: `/v3/projects/fab6800c0a424d719895d1277603a69a` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:02.469738 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fab6800c0a424d719895d1277603a69a)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:02.471660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:02.517594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:02.519175 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fab6800c0a424d719895d1277603a69a', 'request_id': 'req-a4d4188d-9da6-468d-b478-6c00380b1626'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.526096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fab6800c0a424d719895d1277603a69a', 'request_id': 'req-a4d4188d-9da6-468d-b478-6c00380b1626'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.532593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fab6800c0a424d719895d1277603a69a', 'request_id': 'req-a4d4188d-9da6-468d-b478-6c00380b1626'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.545280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] The token cache is being invalidate because project fab6800c0a424d719895d1277603a69a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:02.545569 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.546222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.547689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1424/2828] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:02 2022] DELETE /identity/v3/projects/fab6800c0a424d719895d1277603a69a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:02.551613 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:02.556491 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:02.562262 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343dd997-a38a-4483-989e-d3927e9c00f0 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:02.618304 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:02.618928 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:02.619211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:02.619483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] PATH_INFO: `/v3/users/901bd88097f24e15b4a38f84c26df2dd` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:02.620638 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] RBAC: Authorizing `identity:delete_user(user_id=901bd88097f24e15b4a38f84c26df2dd)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:02.622786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:02.667089 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '901bd88097f24e15b4a38f84c26df2dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.667726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '901bd88097f24e15b4a38f84c26df2dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.674679 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '901bd88097f24e15b4a38f84c26df2dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.675284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '901bd88097f24e15b4a38f84c26df2dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.682118 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-683191b4-f6b8-4261-b628-5c2e189cf522 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '901bd88097f24e15b4a38f84c26df2dd'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:02.698765 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1405/2829] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:02 2022] DELETE /identity/v3/users/901bd88097f24e15b4a38f84c26df2dd => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:02.963119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:02.964663 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:02.978619 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d4188d-9da6-468d-b478-6c00380b1626 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.035177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.035817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.036099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.036374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] PATH_INFO: `/v3/projects/bc686f35b7954a7ca7047211804e2510` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.037483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc686f35b7954a7ca7047211804e2510)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.039939 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.084460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc686f35b7954a7ca7047211804e2510', 'request_id': 'req-89a148ec-4650-49a4-b167-e0c72edeb57f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:03.089930 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc686f35b7954a7ca7047211804e2510', 'request_id': 'req-89a148ec-4650-49a4-b167-e0c72edeb57f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:03.096114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc686f35b7954a7ca7047211804e2510', 'request_id': 'req-89a148ec-4650-49a4-b167-e0c72edeb57f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:03.108714 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] The token cache is being invalidate because project bc686f35b7954a7ca7047211804e2510 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:03.109004 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:03.109625 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-89a148ec-4650-49a4-b167-e0c72edeb57f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:03.111121 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1425/2830] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:02 2022] DELETE /identity/v3/projects/bc686f35b7954a7ca7047211804e2510 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:19:03.116320 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.116607 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.116881 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.136866 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:03.138087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:03.194951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1406/2831] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:03.202898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.203183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.203460 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.223272 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:03.224464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:03.232964 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1426/2832] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:03.236994 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.247627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223737e8-5f9f-4ef9-88d5-98d8e9b6b311 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.252222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.252827 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.253106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.253378 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.254087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:03.254391 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.255977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.267536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1407/2833] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:19:03.271858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.281546 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16145c-bac2-42cf-907a-30a548e1145f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.286182 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.286796 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.287058 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.287332 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.287951 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.290519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.317496 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1427/2834] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/projects => generated 357 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:03.321782 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.331540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a67249-7b4e-42be-bffc-377a5d5a32f0 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.336579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:03.389293 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.389892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.390206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.390483 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.391119 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.393028 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.413734 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1408/2835] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/users => generated 279 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:19:03.418454 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.428613 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8228c2c-1b9b-4956-b004-d02a17b2cf49 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.433327 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.433928 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.434240 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.434516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.435158 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.436742 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.446576 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1428/2836] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:03.450219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.459960 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44664ab7-9644-41f9-901f-be1a20a05015 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.465191 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.465803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.466141 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.466415 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.467062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.468706 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599457ac-2e35-416f-b2c2-7f13a78110e9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.479872 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1409/2837] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:03.483980 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.493919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff65df-ddbb-44c2-afb5-70faab836009 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.498825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.499472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.499754 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.500030 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] PATH_INFO: `/v3/projects/d502a454c162432ab6cb70850ccd85d1/users/16d96d5bae9d4212a1353ac84e75932e/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.514472 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d502a454c162432ab6cb70850ccd85d1, user_id=16d96d5bae9d4212a1353ac84e75932e, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.517163 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.530193 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1429/2838] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:03 2022] PUT /identity/v3/projects/d502a454c162432ab6cb70850ccd85d1/users/16d96d5bae9d4212a1353ac84e75932e/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:03.594097 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.594655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.594655 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.623565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] MFA Rules not processed for user `16d96d5bae9d4212a1353ac84e75932e`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:03.624777 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:03.688523 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1410/2839] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:03.693042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.698064 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:03.758604 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb7d3b-aea5-4583-a390-e004a90dc200 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.763407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.764026 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.764310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.764586 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.765215 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.767184 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.793552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1430/2840] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/projects => generated 355 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:03.797503 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.808278 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3e82c-add2-4a7f-8ae2-0d1827a08e93 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.862405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.863101 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.863433 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.863707 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.864317 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.866237 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.887229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1411/2841] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:03 2022] POST /identity/v3/users => generated 284 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:03.891179 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.902203 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0741ac10-9c65-498b-8f68-202bca142054 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.906821 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.907426 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.907709 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.907984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.908627 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.910245 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.921505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1431/2842] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:03.925379 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.935557 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f8abf-c65c-472f-a9f6-ced2118d06d1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:03.940117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:03.940762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:03.941036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:03.941323 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] PATH_INFO: `/v3/projects/19c9193c30324666a486262d462f1626/users/3b133494e7de4792ac7e3ad5a5e66c08/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:03.943581 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:03.958932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=19c9193c30324666a486262d462f1626, user_id=3b133494e7de4792ac7e3ad5a5e66c08, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:03.962265 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:03.974427 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1412/2843] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:03 2022] PUT /identity/v3/projects/19c9193c30324666a486262d462f1626/users/3b133494e7de4792ac7e3ad5a5e66c08/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:03.978519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:03.988469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d026a-acd8-4803-a9ed-e5a3c8aa1f06 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:04.044188 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:04.044825 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:04.045131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:04.045403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:04.046071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:04.047651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf699326-4637-4d28-8fc2-964a0d510ede admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:04.057681 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1432/2844] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:04.061788 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:04.072618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1a9079-cddb-47b9-838c-6ca454a3b2ef admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:04.077099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:04.077743 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:04.078046 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:04.078316 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] PATH_INFO: `/v3/projects/19c9193c30324666a486262d462f1626/users/3b133494e7de4792ac7e3ad5a5e66c08/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:04.080183 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19c9193c30324666a486262d462f1626, user_id=3b133494e7de4792ac7e3ad5a5e66c08, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:04.082812 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:04.092757 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1413/2845] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:04 2022] PUT /identity/v3/projects/19c9193c30324666a486262d462f1626/users/3b133494e7de4792ac7e3ad5a5e66c08/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:04.102524 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:04.102817 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:04.103097 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:04.132222 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] MFA Rules not processed for user `3b133494e7de4792ac7e3ad5a5e66c08`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:04.134044 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:04.195750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:04.201963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1433/2846] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:04 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:04.213451 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:04.215106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:04.272192 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e9887b-880c-47a7-b178-22c67698ff94 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:04.276380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:04.276991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:04.277253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:04.277518 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:04.286445 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:04.287835 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:04.297627 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1414/2847] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:04 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:04.372388 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:04.382191 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc0c1c-9f06-44ad-99cf-4e6deb8e6886 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:04.386384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:04.386992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:04.387301 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:04.387577 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:04.442490 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:04.443914 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20911c9-734e-4592-af3e-9691c89cbf2d service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:04.453321 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1434/2848] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:04 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:14.293539 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=38,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:14.449237 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=29 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:19.369179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:19.428561 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a11c9-a608-466a-9192-22a33f8faf25 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:19.433621 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:19.434259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:19.434536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:19.434808 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] PATH_INFO: `/v3/users/16d96d5bae9d4212a1353ac84e75932e` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:19.435906 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=16d96d5bae9d4212a1353ac84e75932e)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:19.437795 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:19.481650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16d96d5bae9d4212a1353ac84e75932e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:19.482325 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16d96d5bae9d4212a1353ac84e75932e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:19.489276 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16d96d5bae9d4212a1353ac84e75932e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:19.489880 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16d96d5bae9d4212a1353ac84e75932e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:19.497462 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16d96d5bae9d4212a1353ac84e75932e'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:19.513951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1415/2849] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:19 2022] DELETE /identity/v3/users/16d96d5bae9d4212a1353ac84e75932e => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:19.518501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:19.518981 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:19.519099 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:19.537510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:19.538729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:19.596244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1435/2850] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:19.606588 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:19.608156 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:19.664200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12f269-6a98-4f30-80ad-9e9bba5db6bd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:19.668287 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:19.668884 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:19.669161 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:19.669432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:19.678246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:19.679664 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:19.689603 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1416/2851] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:19.948318 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:19.949744 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:19.963407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a5d1f2-e321-4b75-b0f6-f77579c80d6e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:19.968243 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:19.969623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:19.970123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:19.970403 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] PATH_INFO: `/v3/projects/d502a454c162432ab6cb70850ccd85d1` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:19.971629 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d502a454c162432ab6cb70850ccd85d1)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:19.973462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.020651 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd502a454c162432ab6cb70850ccd85d1', 'request_id': 'req-6770add4-a3ef-45c0-a66f-928cd41d4a75'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.026095 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd502a454c162432ab6cb70850ccd85d1', 'request_id': 'req-6770add4-a3ef-45c0-a66f-928cd41d4a75'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.032081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd502a454c162432ab6cb70850ccd85d1', 'request_id': 'req-6770add4-a3ef-45c0-a66f-928cd41d4a75'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.044136 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] The token cache is being invalidate because project d502a454c162432ab6cb70850ccd85d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:20.044424 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.045042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.046562 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1436/2852] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:19 2022] DELETE /identity/v3/projects/d502a454c162432ab6cb70850ccd85d1 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:19:20.050485 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.060394 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bcb713-361b-4722-86c9-6e738fb4a70f service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.079298 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:20.118939 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.119565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.119838 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.120110 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] PATH_INFO: `/v3/users/3b133494e7de4792ac7e3ad5a5e66c08` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.121473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b133494e7de4792ac7e3ad5a5e66c08)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.123467 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.168843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b133494e7de4792ac7e3ad5a5e66c08'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.169521 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b133494e7de4792ac7e3ad5a5e66c08'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.175742 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b133494e7de4792ac7e3ad5a5e66c08'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.176359 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b133494e7de4792ac7e3ad5a5e66c08'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.183850 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-08af467c-6e59-41bd-8101-7184171a29a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b133494e7de4792ac7e3ad5a5e66c08'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.200826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1417/2853] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:20 2022] DELETE /identity/v3/users/3b133494e7de4792ac7e3ad5a5e66c08 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:20.464848 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.466393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:20.480530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770add4-a3ef-45c0-a66f-928cd41d4a75 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.537729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.538432 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.538662 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.538935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] PATH_INFO: `/v3/projects/19c9193c30324666a486262d462f1626` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.540668 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19c9193c30324666a486262d462f1626)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.542606 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.571868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:20.589423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19c9193c30324666a486262d462f1626', 'request_id': 'req-48857727-3c95-4b89-89c4-52b265bc3d40'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.594927 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19c9193c30324666a486262d462f1626', 'request_id': 'req-48857727-3c95-4b89-89c4-52b265bc3d40'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.601002 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19c9193c30324666a486262d462f1626', 'request_id': 'req-48857727-3c95-4b89-89c4-52b265bc3d40'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.615122 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] The token cache is being invalidate because project 19c9193c30324666a486262d462f1626 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:20.615407 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.616021 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-48857727-3c95-4b89-89c4-52b265bc3d40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:20.617568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1437/2854] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:20 2022] DELETE /identity/v3/projects/19c9193c30324666a486262d462f1626 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:20.622727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.623201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.623315 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.642261 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:20.643490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:20.701170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1418/2855] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:20.710186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.710471 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.710745 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.735597 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:20.736824 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:20.744751 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1438/2856] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:20.749469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.758962 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60522d2b-5032-4515-8f0b-e216c5d05653 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.763473 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.764067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.764345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.764617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.765257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:20.765556 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.767164 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.778977 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1419/2857] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:19:20.783096 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.792929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f101e81-de8a-423b-a3f8-5a00bc705ebf None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.797582 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.798207 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.798493 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.798764 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.799399 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.801280 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.829149 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1439/2858] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:20 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:20.833360 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.843185 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c926eb9-7032-459d-a8f1-0092815e8499 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.869257 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:20.900746 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.901281 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.901559 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.901830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.902476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.904363 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.927206 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1420/2859] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:20 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:20.931846 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.941871 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534c6e1-e1c1-4e07-ab17-8b2450fd7b51 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.946484 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.947077 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.947352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.947620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.948252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.949832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.959539 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1440/2860] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:20.963399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:20.973075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d7abe-239d-4751-9afb-7641340bbbf7 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:20.978330 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:20.978922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:20.979201 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:20.979470 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:20.980108 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:20.981865 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42d6077-85af-445a-9825-50dec799fff3 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:20.992230 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1421/2861] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:20.997102 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.006875 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5654f9c-2491-49f7-a8cb-9ee6210084a1 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.011499 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.012142 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.012418 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.012686 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] PATH_INFO: `/v3/projects/6cff23ddc78a400bafc9a2155fe6400c/users/4702c6275da3438999d65c6a54378fe5/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.027891 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6cff23ddc78a400bafc9a2155fe6400c, user_id=4702c6275da3438999d65c6a54378fe5, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.030556 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.043785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1441/2862] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:20 2022] PUT /identity/v3/projects/6cff23ddc78a400bafc9a2155fe6400c/users/4702c6275da3438999d65c6a54378fe5/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:19:21.054113 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.054399 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.054670 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.082922 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] MFA Rules not processed for user `4702c6275da3438999d65c6a54378fe5`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:21.084153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:21.146647 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1422/2863] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:21 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:21.151940 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.211932 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071228f8-90f6-4ba8-a49c-bec9b77bc86c admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.216507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.217101 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.217379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.217646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.218291 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.220161 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.246596 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1442/2864] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:21 2022] POST /identity/v3/projects => generated 363 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:21.251080 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.262387 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b011a4b3-16c4-4bae-abcf-ec7784a5d9fd None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.317519 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.318150 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.318431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.318703 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.321832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.341897 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1423/2865] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:21 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:19:21.345922 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.356392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de6ab0d-c166-4a11-a361-f1993a01dd82 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.360957 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.361555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.361834 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.362138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.362805 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.364377 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.375729 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1443/2866] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:21.379719 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.390283 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e925ea-e650-4244-b79e-4ddfe062f76c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.394834 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.395484 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.395762 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.396036 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] PATH_INFO: `/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a/users/e3dd0c4c4a004fcb95e137055d326ac4/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.410417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=56a0acabf9684c0682895c0b7b9a9a3a, user_id=e3dd0c4c4a004fcb95e137055d326ac4, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.412985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.425076 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1424/2867] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:21 2022] PUT /identity/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a/users/e3dd0c4c4a004fcb95e137055d326ac4/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:19:21.428895 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.439354 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1ddb9-a2c1-47ca-afb9-a2d1d4a5b48a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.459087 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:21.496196 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.496790 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.497066 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.497334 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.497960 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.499575 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbba1d7-e026-4c2e-8ce6-25dd2e241ee5 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.509630 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1444/2868] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:21 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:19:21.513840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.519476 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:21.525562 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315da2bf-8bb7-42d3-9603-c0d60fbd3fbf admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.530508 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.531153 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.531432 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.531702 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] PATH_INFO: `/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a/users/e3dd0c4c4a004fcb95e137055d326ac4/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.533789 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56a0acabf9684c0682895c0b7b9a9a3a, user_id=e3dd0c4c4a004fcb95e137055d326ac4, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.536464 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.546471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1425/2869] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:21 2022] PUT /identity/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a/users/e3dd0c4c4a004fcb95e137055d326ac4/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:21.558092 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.558601 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.558719 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.586435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] MFA Rules not processed for user `e3dd0c4c4a004fcb95e137055d326ac4`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:21.587647 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:21.654612 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1445/2870] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:21 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:21.666239 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.667879 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:21.723617 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acc7aa0-7783-4fb1-aa4a-1dbb11c3c470 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.727665 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.728254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.728530 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.728799 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.737661 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.739121 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.749244 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1426/2871] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:21 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:21.821865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:21.831608 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6916dfc6-f477-4167-aa22-7c8ce9dfab9c None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:21.835230 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:21.835833 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:21.836115 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:21.836430 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:21.894768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:21.896214 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65568d1-e153-47c2-a164-44ccbd4e145f service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:21.905138 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1446/2872] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:21 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:24.611009 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:24.671225 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b089a4-431c-491c-84a8-f3a6504d86dd service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:24.675701 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:24.676348 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:24.676626 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:24.676898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] PATH_INFO: `/v3/users/4702c6275da3438999d65c6a54378fe5` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:24.678029 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4702c6275da3438999d65c6a54378fe5)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:24.679933 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:24.724200 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4702c6275da3438999d65c6a54378fe5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:24.724843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4702c6275da3438999d65c6a54378fe5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:24.731639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4702c6275da3438999d65c6a54378fe5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:24.732246 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4702c6275da3438999d65c6a54378fe5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:24.739040 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4702c6275da3438999d65c6a54378fe5'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:24.757107 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1427/2873] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:24 2022] DELETE /identity/v3/users/4702c6275da3438999d65c6a54378fe5 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:24.761894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:24.762458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:24.762458 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:24.782389 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:24.783571 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:24.822482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:24.841941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1447/2874] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:24.851480 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:24.852893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:24.866211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:24.910605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4453e9-c39d-466f-8a58-a3e060d5947c admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:24.914253 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:24.914855 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:24.915160 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:24.915436 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:24.924219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:24.925645 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:24.934893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1428/2875] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:25.194464 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:25.195898 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:25.209945 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952794f4-ef74-4110-b276-f3c43333fdd6 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:25.214865 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:25.215513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:25.215789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:25.216060 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] PATH_INFO: `/v3/projects/6cff23ddc78a400bafc9a2155fe6400c` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:25.217183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6cff23ddc78a400bafc9a2155fe6400c)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:25.220114 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:25.267152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6cff23ddc78a400bafc9a2155fe6400c', 'request_id': 'req-9fb85dcc-49a0-4d4d-8371-b25088c89865'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.274409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6cff23ddc78a400bafc9a2155fe6400c', 'request_id': 'req-9fb85dcc-49a0-4d4d-8371-b25088c89865'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.280547 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6cff23ddc78a400bafc9a2155fe6400c', 'request_id': 'req-9fb85dcc-49a0-4d4d-8371-b25088c89865'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.292441 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] The token cache is being invalidate because project 6cff23ddc78a400bafc9a2155fe6400c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:25.292723 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.293349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.294860 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1448/2876] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:25 2022] DELETE /identity/v3/projects/6cff23ddc78a400bafc9a2155fe6400c => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:25.298952 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:25.308988 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33753216-0415-4d1a-ab72-684ace08e0d4 service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:25.364950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:25.365495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:25.365767 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:25.366059 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] PATH_INFO: `/v3/users/e3dd0c4c4a004fcb95e137055d326ac4` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:25.367154 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3dd0c4c4a004fcb95e137055d326ac4)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:25.369691 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:25.412534 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3dd0c4c4a004fcb95e137055d326ac4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.413175 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3dd0c4c4a004fcb95e137055d326ac4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.422853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3dd0c4c4a004fcb95e137055d326ac4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.423469 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3dd0c4c4a004fcb95e137055d326ac4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.430421 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-447bf017-f7c2-4114-b060-610ce1f73418 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3dd0c4c4a004fcb95e137055d326ac4'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.447675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1429/2877] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:25 2022] DELETE /identity/v3/users/e3dd0c4c4a004fcb95e137055d326ac4 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:25.707368 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:25.708868 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:25.762130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb85dcc-49a0-4d4d-8371-b25088c89865 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:25.816869 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:25.817506 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:25.817782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:25.818089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] PATH_INFO: `/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:25.819211 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56a0acabf9684c0682895c0b7b9a9a3a)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:25.822071 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:25.858978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:25.868935 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56a0acabf9684c0682895c0b7b9a9a3a', 'request_id': 'req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.874799 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56a0acabf9684c0682895c0b7b9a9a3a', 'request_id': 'req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.881231 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56a0acabf9684c0682895c0b7b9a9a3a', 'request_id': 'req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.892733 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] The token cache is being invalidate because project 56a0acabf9684c0682895c0b7b9a9a3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:25.892959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.893568 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-0ad5e8dd-93e7-4b6d-a278-fb627972e4e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:25.895057 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1449/2878] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:25 2022] DELETE /identity/v3/projects/56a0acabf9684c0682895c0b7b9a9a3a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:25.900177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:25.900658 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:25.900763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:25.919568 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:25.920741 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:25.949337 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:25.979867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1430/2879] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:25.987934 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:25.988250 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:25.988553 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.008443 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:26.010300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:26.018081 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1450/2880] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:26.022226 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.031663 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f601b26a-7c74-4c34-8a2b-4969c1aa4bff None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.036147 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.036744 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.037021 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.037292 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] PATH_INFO: `/v3/domains` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.038077 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81306) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:26.038405 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.039986 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.051271 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1431/2881] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:19:26.055312 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.064941 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c006db-cc60-4341-aa8b-4b9d4c78b7f3 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.069492 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.070152 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.070434 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.070707 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.071337 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.073808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.099646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1451/2882] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/projects => generated 381 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:26.104173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.113901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a02699d-9b56-497a-9858-a3213af35d2e admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.169318 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.169843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.170162 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.170458 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.171074 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.172959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.193907 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1432/2883] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/users => generated 291 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:26.197963 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.208187 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76236d98-ed1f-49bc-91a4-e0d4120b81f8 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.212789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.213384 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.213660 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.213929 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.214600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.216181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.225854 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1452/2884] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:26.230048 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.239654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadaa980-5dde-4057-8ea9-9c1ce0c88412 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.244783 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.245368 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.246355 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.246629 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.247268 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.248847 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a78f8e3-4642-4fb1-ae26-607e2070d94b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.259877 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1433/2885] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:26.264262 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.274748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d925-5fbf-4fae-aa10-c0dbe94cd619 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.279519 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.280183 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.280462 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.280739 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] PATH_INFO: `/v3/projects/89982cd6ef024dd6af4168b6046cdaaa/users/2aa4f3f4930a4bc98ccf756f1fb028c4/roles/770c965de4374724a8a6de8042997c97` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.296696 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89982cd6ef024dd6af4168b6046cdaaa, user_id=2aa4f3f4930a4bc98ccf756f1fb028c4, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.299416 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.312198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1453/2886] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:26 2022] PUT /identity/v3/projects/89982cd6ef024dd6af4168b6046cdaaa/users/2aa4f3f4930a4bc98ccf756f1fb028c4/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:26.322600 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.322954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.323249 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.349770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] MFA Rules not processed for user `2aa4f3f4930a4bc98ccf756f1fb028c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:26.351025 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:26.414540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1434/2887] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:26.419783 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.479816 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b3afe6-5487-409b-8a93-4da44d3616a9 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.484362 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.484959 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.485236 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.485507 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.486168 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.488042 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.513736 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1454/2888] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/projects => generated 381 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:19:26.517527 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.528450 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b70191-1a3e-4886-b4a7-49259f3274ae None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.553803 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:26.584697 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.585331 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.585590 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.585868 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.586540 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.588959 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.610362 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1435/2889] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/users => generated 297 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:26.614812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.625372 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a25f57-f2b8-4cfe-9c51-4878d2ea2e64 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.629969 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.630593 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.630873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.631148 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.631789 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.633386 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.643788 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1455/2890] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:26.647802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.658017 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc83be-6111-4939-9ee3-22a5d06898b2 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.662537 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.663172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.663446 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.663712 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] PATH_INFO: `/v3/projects/c994c0c62bc14cdb990704f80edd25f9/users/a9ca81a8d8624f94b1a0c092ab7ded31/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.678704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c994c0c62bc14cdb990704f80edd25f9, user_id=a9ca81a8d8624f94b1a0c092ab7ded31, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.681250 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.694049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1436/2891] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:26 2022] PUT /identity/v3/projects/c994c0c62bc14cdb990704f80edd25f9/users/a9ca81a8d8624f94b1a0c092ab7ded31/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 11:19:26.697838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.707893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0113-d64b-43b3-9ebf-9903b651682a admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.739873 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:26.765851 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.766533 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.766778 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.767039 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.767682 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.769271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18df613-03a6-4b8f-8049-28f2fa4e4e75 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.779477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1456/2892] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:26.783618 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.794309 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff37c772-ea44-4bbd-958f-6ea6cf3d0c16 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:26.798908 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:26.799549 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.799824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.800093 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] PATH_INFO: `/v3/projects/c994c0c62bc14cdb990704f80edd25f9/users/a9ca81a8d8624f94b1a0c092ab7ded31/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.801999 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c994c0c62bc14cdb990704f80edd25f9, user_id=a9ca81a8d8624f94b1a0c092ab7ded31, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:26.804667 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:26.814641 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1437/2893] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:26 2022] PUT /identity/v3/projects/c994c0c62bc14cdb990704f80edd25f9/users/a9ca81a8d8624f94b1a0c092ab7ded31/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:26.826421 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:26.826712 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:26.826987 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:26.854813 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] MFA Rules not processed for user `a9ca81a8d8624f94b1a0c092ab7ded31`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:26.856005 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:26.922513 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1457/2894] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:26 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:26.934716 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:26.936417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:27.009428 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4d51d8-397c-4db0-8158-b1f97227139b admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.013014 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.013615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.013895 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.014211 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.022875 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.024300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.034256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1438/2895] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:26 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:27.341752 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:27.350676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6c6c6f-06d4-415f-98ed-ae78ee0fb283 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.355151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9ca81a8d8624f94b1a0c092ab7ded31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c994c0c62bc14cdb990704f80edd25f9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.355748 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.356034 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.356311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.356947 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:27.357251 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.358863 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.370555 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1458/2896] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:19:27.374786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:27.383873 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a919b4e-cb4d-409c-9f38-ed8f6d48b7e8 service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.389043 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9ca81a8d8624f94b1a0c092ab7ded31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c994c0c62bc14cdb990704f80edd25f9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.389639 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.389925 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.390231 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.390867 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.392763 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.420511 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1439/2897] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:27 2022] POST /identity/v3/projects => generated 354 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:27.424551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:27.434311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaeb10d-dc6d-46a8-8447-b2e1a0d8cdf4 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.488530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9ca81a8d8624f94b1a0c092ab7ded31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c994c0c62bc14cdb990704f80edd25f9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.489129 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.489406 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.489675 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.490344 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:27.490646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.492217 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.504137 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1459/2898] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 11:19:27.541117 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:27.551264 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f0e7bf-5fa0-4b89-81b2-abd404f1c524 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.555903 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9ca81a8d8624f94b1a0c092ab7ded31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c994c0c62bc14cdb990704f80edd25f9', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.556554 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.556832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.557254 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] PATH_INFO: `/v3/projects/0abe6d1fefdf4c3781b6c27ffe9148c5` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.558419 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorizing `identity:delete_project(project_id=0abe6d1fefdf4c3781b6c27ffe9148c5)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.560301 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.600932 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:INSERT=6,keystone:DELETE=3 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:27.608373 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0abe6d1fefdf4c3781b6c27ffe9148c5', 'request_id': 'req-3f5fcce5-6d38-4011-b01e-a444d3182314'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.614091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0abe6d1fefdf4c3781b6c27ffe9148c5', 'request_id': 'req-3f5fcce5-6d38-4011-b01e-a444d3182314'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.620217 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0abe6d1fefdf4c3781b6c27ffe9148c5', 'request_id': 'req-3f5fcce5-6d38-4011-b01e-a444d3182314'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.632370 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] The token cache is being invalidate because project 0abe6d1fefdf4c3781b6c27ffe9148c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:27.632662 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.633288 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-3f5fcce5-6d38-4011-b01e-a444d3182314 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.635455 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1440/2899] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:27 2022] DELETE /identity/v3/projects/0abe6d1fefdf4c3781b6c27ffe9148c5 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:27.819858 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:27.822093 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:27.864078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=100 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:27.887369 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d278fbb-f7d5-42fb-b019-fd6c884ddd24 tempest-VolumeQuotaClassesTest-1831314726 tempest-VolumeQuotaClassesTest-1831314726-project-admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:27.940356 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:27.940978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:27.941255 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:27.941523 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] PATH_INFO: `/v3/users/2aa4f3f4930a4bc98ccf756f1fb028c4` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:27.942649 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2aa4f3f4930a4bc98ccf756f1fb028c4)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:27.944552 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:27.989079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2aa4f3f4930a4bc98ccf756f1fb028c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.996893 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2aa4f3f4930a4bc98ccf756f1fb028c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.997510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2aa4f3f4930a4bc98ccf756f1fb028c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:27.998166 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2aa4f3f4930a4bc98ccf756f1fb028c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.005019 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2aa4f3f4930a4bc98ccf756f1fb028c4'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.022194 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1460/2900] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:27 2022] DELETE /identity/v3/users/2aa4f3f4930a4bc98ccf756f1fb028c4 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:28.026623 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:28.027103 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:28.027220 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:28.045382 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:28.046666 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:28.105199 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1441/2901] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:28.114698 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:28.116279 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:28.174119 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df88564-0808-4d2d-888b-4d1359a410a4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:28.177755 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:28.178422 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:28.178750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:28.179048 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:28.188622 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:28.190062 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:28.199861 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1461/2902] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:28.465683 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:28.467227 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:28.480624 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e77241b-914a-4bd3-8e84-d84af5464abb None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:28.485171 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:28.486260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:28.486260 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:28.486501 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] PATH_INFO: `/v3/projects/89982cd6ef024dd6af4168b6046cdaaa` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:28.487541 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89982cd6ef024dd6af4168b6046cdaaa)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:28.489447 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:28.535650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89982cd6ef024dd6af4168b6046cdaaa', 'request_id': 'req-eb1f782c-361a-40bb-ba65-7dea235999b1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.541826 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89982cd6ef024dd6af4168b6046cdaaa', 'request_id': 'req-eb1f782c-361a-40bb-ba65-7dea235999b1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.547893 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89982cd6ef024dd6af4168b6046cdaaa', 'request_id': 'req-eb1f782c-361a-40bb-ba65-7dea235999b1'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.560186 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] The token cache is being invalidate because project 89982cd6ef024dd6af4168b6046cdaaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:28.560494 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.561084 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.562552 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1442/2903] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:28 2022] DELETE /identity/v3/projects/89982cd6ef024dd6af4168b6046cdaaa => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:28.566705 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:28.576455 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca677b3-33b3-4e02-bbee-165a0cc613bd service neutron] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:28.580548 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:28.634855 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:28.635482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:28.635760 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:28.636032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] PATH_INFO: `/v3/users/a9ca81a8d8624f94b1a0c092ab7ded31` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:28.637130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9ca81a8d8624f94b1a0c092ab7ded31)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:28.639089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:28.683870 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9ca81a8d8624f94b1a0c092ab7ded31'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.690126 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9ca81a8d8624f94b1a0c092ab7ded31'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.690725 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9ca81a8d8624f94b1a0c092ab7ded31'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.691315 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9ca81a8d8624f94b1a0c092ab7ded31'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.698078 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-1fdcc545-1e18-4a41-bf86-9abe477c75f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9ca81a8d8624f94b1a0c092ab7ded31'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:28.715702 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1462/2904] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:28 2022] DELETE /identity/v3/users/a9ca81a8d8624f94b1a0c092ab7ded31 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:28.981923 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:28.983431 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:28.997943 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f782c-361a-40bb-ba65-7dea235999b1 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.047453 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:29.057465 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.058128 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.058440 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.058710 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] PATH_INFO: `/v3/projects/c994c0c62bc14cdb990704f80edd25f9` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.059837 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] RBAC: Authorizing `identity:delete_project(project_id=c994c0c62bc14cdb990704f80edd25f9)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.061735 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.106075 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c994c0c62bc14cdb990704f80edd25f9', 'request_id': 'req-7804ad5c-9169-4c17-9863-db7e3a0dcbde'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:29.111366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c994c0c62bc14cdb990704f80edd25f9', 'request_id': 'req-7804ad5c-9169-4c17-9863-db7e3a0dcbde'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:29.117219 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c994c0c62bc14cdb990704f80edd25f9', 'request_id': 'req-7804ad5c-9169-4c17-9863-db7e3a0dcbde'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:29.129745 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] The token cache is being invalidate because project c994c0c62bc14cdb990704f80edd25f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81306) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:29.130063 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:29.130731 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-7804ad5c-9169-4c17-9863-db7e3a0dcbde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:29.132173 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1443/2905] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:28 2022] DELETE /identity/v3/projects/c994c0c62bc14cdb990704f80edd25f9 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:29.137666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.138177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.138310 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.158130 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:29.159317 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:29.216828 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1463/2906] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:29.224605 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.224830 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.225099 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.244284 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:29.245471 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:29.253802 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1444/2907] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 11:19:29.258003 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.268689 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e55a2c-5b07-4131-9b9b-928d69b44b87 None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.273284 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.273890 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.274212 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.275103 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] PATH_INFO: `/v3/domains` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.275750 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81305) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 11:19:29.276053 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.277644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.289358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1464/2908] 10.222.0.46 () {58 vars in 1153 bytes} [Fri Oct 28 11:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 11:19:29.293584 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.302848 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2dd8a5-e8fa-4cca-9785-764dc8f517a8 None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.307429 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.308018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.308295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.308565 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.309194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.311095 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.339259 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1445/2909] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:29.343393 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.353084 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113b3c0c-5c10-453c-8342-61f88c0ff052 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.361830 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:29.410482 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.411107 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.411385 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.411656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.412349 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.414271 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.434721 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1465/2910] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:29.438786 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.449632 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ad6375-2be7-4166-ba66-788fa28be2fa admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.454255 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.454851 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.455127 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.455397 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.456035 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.457616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.467840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1446/2911] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:29.471691 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.482079 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e09bc6-d399-4e66-8066-9f9e523dce4e admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.486551 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.487162 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.487423 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.487694 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.488345 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.489998 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6c4503-ade6-4464-9329-99326b77bdfc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.501210 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1466/2912] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 11:19:29.505181 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.515653 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4da049-538c-42da-beea-df7d2cf8734a admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.520157 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.520794 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.521071 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.521342 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] PATH_INFO: `/v3/projects/75f681e0dd5149b9a1b92906ff9e1d62/users/c0898160e6da4e0e9269246cfbed9e9a/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.535434 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=75f681e0dd5149b9a1b92906ff9e1d62, user_id=c0898160e6da4e0e9269246cfbed9e9a, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.538087 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.550538 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1447/2913] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:29 2022] PUT /identity/v3/projects/75f681e0dd5149b9a1b92906ff9e1d62/users/c0898160e6da4e0e9269246cfbed9e9a/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:29.561232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.561516 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.561786 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.590151 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] MFA Rules not processed for user `c0898160e6da4e0e9269246cfbed9e9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:29.591379 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:29.654261 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1467/2914] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:29.658675 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.718612 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96f6bd8-7513-4a2c-9def-5267a9a902eb admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.723195 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.723792 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.724073 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.724345 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.724974 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.726901 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.752817 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1448/2915] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/projects => generated 387 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 11:19:29.756923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.768159 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0bef11-ddc2-4520-930b-ec1cb9c38b1b None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.825089 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.825687 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.825961 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.826257 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.826876 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.828777 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.848992 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1468/2916] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:29.853704 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.860234 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:29.864302 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693d97c-65e5-489c-9c62-d72c94a5a9be admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.870686 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.871544 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.871948 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.872380 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.873018 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.874726 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.885312 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1449/2917] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:29.889140 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.898991 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff7bb4-0983-4f3e-82f3-2248bf6970c4 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.903559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.904134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.904409 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.904680 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.905307 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.906916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c8059b-d199-425e-bb36-dc91db5f9f82 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.911974 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:29.918559 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1469/2918] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:29.922065 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:29.932466 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc16e-283d-47ab-b9b3-66017cf160fd admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:29.937011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:29.937649 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.937930 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.938238 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] PATH_INFO: `/v3/projects/170eb1f809bd4b0c8482049b847695a2/users/cdd0b99107c14fccbc5a8650e2a635a8/roles/770c965de4374724a8a6de8042997c97` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:29.953184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=170eb1f809bd4b0c8482049b847695a2, user_id=cdd0b99107c14fccbc5a8650e2a635a8, role_id=770c965de4374724a8a6de8042997c97)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:29.955831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:29.968067 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1450/2919] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:29 2022] PUT /identity/v3/projects/170eb1f809bd4b0c8482049b847695a2/users/cdd0b99107c14fccbc5a8650e2a635a8/roles/770c965de4374724a8a6de8042997c97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:29.979676 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:29.979955 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:29.980232 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.008674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] MFA Rules not processed for user `cdd0b99107c14fccbc5a8650e2a635a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:30.009838 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:30.071923 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1470/2920] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:29 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:30.076346 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.137832 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ea799e-1a99-4b53-a2fc-8c90a770a3a9 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.142452 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.143049 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.143324 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.143594 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] PATH_INFO: `/v3/projects` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.144222 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.146132 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.172536 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1451/2921] 10.222.0.46 () {60 vars in 1153 bytes} [Fri Oct 28 11:19:30 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:30.176618 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.188267 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4172e0-c4fb-404a-9402-1ec3fadaad2e None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.242602 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.243199 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.243510 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.243787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] PATH_INFO: `/v3/users` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.244412 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.247018 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.267785 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1471/2922] 10.222.0.46 () {60 vars in 1144 bytes} [Fri Oct 28 11:19:30 2022] POST /identity/v3/users => generated 299 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 11:19:30.271892 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.283985 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0bca05-2e04-485d-ada8-55809a92c733 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.288630 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.289228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.289505 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.289774 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.291229 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.292840 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.303122 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1452/2923] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:30.307382 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.317177 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0a9c04-d1d7-47ab-b827-2cb398a4affc admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.321812 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.322489 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.322768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.323038 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] PATH_INFO: `/v3/projects/58e7c57cf57a44a990735d6b86d4e736/users/18c9e0a217be4f1abfdaa20e0adcf229/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.337829 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] RBAC: Authorizing `identity:create_grant(project_id=58e7c57cf57a44a990735d6b86d4e736, user_id=18c9e0a217be4f1abfdaa20e0adcf229, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.340570 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.353984 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1472/2924] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:30 2022] PUT /identity/v3/projects/58e7c57cf57a44a990735d6b86d4e736/users/18c9e0a217be4f1abfdaa20e0adcf229/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:30.358106 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.367921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be28a9c-303f-488c-8111-bc70ed34aa47 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.424212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.424815 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.425096 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.425369 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.426026 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.427616 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d520d8f-56c8-48f2-bdb9-2f1cbc938f9d admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.436936 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1453/2925] 10.222.0.46 () {58 vars in 1122 bytes} [Fri Oct 28 11:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 11:19:30.441216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.451286 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5b02a0-2ec0-4a39-8a75-c4c27fc5edda admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.455841 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.456488 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] REQUEST_METHOD: `PUT` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.456762 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.457029 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] PATH_INFO: `/v3/projects/58e7c57cf57a44a990735d6b86d4e736/users/18c9e0a217be4f1abfdaa20e0adcf229/roles/219649f17b76446aa279da8fc7dc40e7` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.458944 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58e7c57cf57a44a990735d6b86d4e736, user_id=18c9e0a217be4f1abfdaa20e0adcf229, role_id=219649f17b76446aa279da8fc7dc40e7)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.461685 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.472374 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1473/2926] 10.222.0.46 () {60 vars in 1483 bytes} [Fri Oct 28 11:19:30 2022] PUT /identity/v3/projects/58e7c57cf57a44a990735d6b86d4e736/users/18c9e0a217be4f1abfdaa20e0adcf229/roles/219649f17b76446aa279da8fc7dc40e7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:30.483207 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280d3082-5b81-4d51-994b-65739b04142c None None] REQUEST_METHOD: `POST` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.483497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280d3082-5b81-4d51-994b-65739b04142c None None] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.483770 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280d3082-5b81-4d51-994b-65739b04142c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.511853 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.auth.core [None req-280d3082-5b81-4d51-994b-65739b04142c None None] MFA Rules not processed for user `18c9e0a217be4f1abfdaa20e0adcf229`. Rule list: `[]` (Enabled: `True`). {{(pid=81306) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:30.513057 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-280d3082-5b81-4d51-994b-65739b04142c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:30.542056 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:30.581863 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1454/2927] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:30 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:30.592538 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:30.594300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:30.640644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:30.653135 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a31d5-9c2a-4762-8b97-82d884c9e378 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:30.656768 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:30.657367 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:30.657644 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:30.657916 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:30.667475 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:30.668894 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:30.678205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1474/2928] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:30 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:31.172580 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280d3082-5b81-4d51-994b-65739b04142c None None] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:31.173991 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-280d3082-5b81-4d51-994b-65739b04142c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:31.185615 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280d3082-5b81-4d51-994b-65739b04142c None None] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:31.189092 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:31.189678 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:31.189951 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:31.190256 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:31.245816 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:31.247602 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:31.256654 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1455/2929] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 11:19:35.255530 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:35.264483 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7f9959-235e-4441-bce7-fc86c744a40e service cinder] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:35.268181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2297b30fe14d4de8a3f39e247ed59dfb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:35.268808 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:35.269086 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:35.269359 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:35.325743 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:35.327198 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb564db-be41-40ee-8672-9dab1b658542 service cinder] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:35.335853 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1475/2930] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:35 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:37.376300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:37.437477 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2aaf85-e1f5-4375-ab05-95f44993ffeb service cinder] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:37.441909 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:37.442548 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:37.442823 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:37.443091 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] PATH_INFO: `/v3/users/c0898160e6da4e0e9269246cfbed9e9a` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:37.444177 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0898160e6da4e0e9269246cfbed9e9a)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:37.446112 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:37.490722 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0898160e6da4e0e9269246cfbed9e9a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:37.491366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0898160e6da4e0e9269246cfbed9e9a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:37.499172 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0898160e6da4e0e9269246cfbed9e9a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:37.499814 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0898160e6da4e0e9269246cfbed9e9a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:37.507221 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0898160e6da4e0e9269246cfbed9e9a'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:37.509179 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:37.526640 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1456/2931] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:37 2022] DELETE /identity/v3/users/c0898160e6da4e0e9269246cfbed9e9a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:37.531134 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] REQUEST_METHOD: `POST` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:37.531620 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:37.531770 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:37.550645 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.auth.core [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] MFA Rules not processed for user `79c132c0cbf74b0899faa3036130f049`. Rule list: `[]` (Enabled: `True`). {{(pid=81305) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 11:19:37.551832 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:37.608435 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1476/2932] 10.222.0.46 () {58 vars in 958 bytes} [Fri Oct 28 11:19:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 11:19:37.618062 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:37.620064 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:37.674831 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4c978-0a6e-482d-93ff-b971453e7163 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:37.678366 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0287d4acc8f548c482ad40bd61b1081d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:37.678954 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:37.679228 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:37.679495 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:37.688843 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:37.690280 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:37.699497 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1457/2933] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:37.958594 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:37.960131 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:37.974123 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7f39e-3d4b-445d-987e-d9abb689a35f None None] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:37.979358 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:37.980000 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:37.980311 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:37.980585 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] PATH_INFO: `/v3/projects/75f681e0dd5149b9a1b92906ff9e1d62` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:37.981711 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] RBAC: Authorizing `identity:delete_project(project_id=75f681e0dd5149b9a1b92906ff9e1d62)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:37.983674 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:38.029300 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75f681e0dd5149b9a1b92906ff9e1d62', 'request_id': 'req-08924d4b-7054-4c59-a5e5-a92707beacdd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.034656 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75f681e0dd5149b9a1b92906ff9e1d62', 'request_id': 'req-08924d4b-7054-4c59-a5e5-a92707beacdd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.041205 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75f681e0dd5149b9a1b92906ff9e1d62', 'request_id': 'req-08924d4b-7054-4c59-a5e5-a92707beacdd'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.053836 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] The token cache is being invalidate because project 75f681e0dd5149b9a1b92906ff9e1d62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:38.054155 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.054782 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.056352 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1477/2934] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:37 2022] DELETE /identity/v3/projects/75f681e0dd5149b9a1b92906ff9e1d62 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:38.060285 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:38.069724 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab63e5a-de14-46ba-af41-73a570300c2a service neutron] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:38.124898 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:38.125520 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:38.125824 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:38.126165 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] PATH_INFO: `/v3/users/cdd0b99107c14fccbc5a8650e2a635a8` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:38.127277 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdd0b99107c14fccbc5a8650e2a635a8)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:38.129170 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:38.173194 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdd0b99107c14fccbc5a8650e2a635a8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.174002 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdd0b99107c14fccbc5a8650e2a635a8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.181579 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdd0b99107c14fccbc5a8650e2a635a8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.182241 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdd0b99107c14fccbc5a8650e2a635a8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.189212 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdd0b99107c14fccbc5a8650e2a635a8'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.206650 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1458/2935] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:38 2022] DELETE /identity/v3/users/cdd0b99107c14fccbc5a8650e2a635a8 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:38.470469 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:38.472154 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:38.485646 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08924d4b-7054-4c59-a5e5-a92707beacdd admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:38.489419 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:38.544316 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:38.544943 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:38.545216 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:38.545487 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] PATH_INFO: `/v3/projects/170eb1f809bd4b0c8482049b847695a2` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:38.546617 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] RBAC: Authorizing `identity:delete_project(project_id=170eb1f809bd4b0c8482049b847695a2)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:38.548522 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:38.595361 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '170eb1f809bd4b0c8482049b847695a2', 'request_id': 'req-24d1159c-9849-4096-8edd-7dc6813c102f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.600837 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '170eb1f809bd4b0c8482049b847695a2', 'request_id': 'req-24d1159c-9849-4096-8edd-7dc6813c102f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.606915 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '170eb1f809bd4b0c8482049b847695a2', 'request_id': 'req-24d1159c-9849-4096-8edd-7dc6813c102f'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.619600 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] The token cache is being invalidate because project 170eb1f809bd4b0c8482049b847695a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:38.619888 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.620505 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.622278 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1478/2936] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:38 2022] DELETE /identity/v3/projects/170eb1f809bd4b0c8482049b847695a2 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:38.626279 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:38.627727 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:38.637361 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:38.641329 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdfd8e3-bbfe-48e3-8dca-d8a7241b3923 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:38.699327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:38.699950 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:38.700224 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:38.700490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] PATH_INFO: `/v3/users/18c9e0a217be4f1abfdaa20e0adcf229` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:38.701578 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18c9e0a217be4f1abfdaa20e0adcf229)` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:38.703532 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:38.748295 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18c9e0a217be4f1abfdaa20e0adcf229'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.748921 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18c9e0a217be4f1abfdaa20e0adcf229'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.755769 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18c9e0a217be4f1abfdaa20e0adcf229'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.756558 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18c9e0a217be4f1abfdaa20e0adcf229'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.764011 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.notifications [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18c9e0a217be4f1abfdaa20e0adcf229'} {{(pid=81306) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:38.781184 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1459/2937] 10.222.0.46 () {58 vars in 1224 bytes} [Fri Oct 28 11:19:38 2022] DELETE /identity/v3/users/18c9e0a217be4f1abfdaa20e0adcf229 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:39.046448 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:39.047919 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:39.061666 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d1159c-9849-4096-8edd-7dc6813c102f admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:39.117706 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '79c132c0cbf74b0899faa3036130f049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2b351b41bbb64be2822e6354d987b945', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:39.118378 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:39.119202 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:39.119477 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] PATH_INFO: `/v3/projects/58e7c57cf57a44a990735d6b86d4e736` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:39.120590 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58e7c57cf57a44a990735d6b86d4e736)` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:39.122787 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:39.161623 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:39.170497 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58e7c57cf57a44a990735d6b86d4e736', 'request_id': 'req-49032513-154b-431b-b629-3a21e4adf350'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:39.175978 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58e7c57cf57a44a990735d6b86d4e736', 'request_id': 'req-49032513-154b-431b-b629-3a21e4adf350'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:39.182665 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58e7c57cf57a44a990735d6b86d4e736', 'request_id': 'req-49032513-154b-431b-b629-3a21e4adf350'} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:39.194252 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] The token cache is being invalidate because project 58e7c57cf57a44a990735d6b86d4e736 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81305) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 11:19:39.194535 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:39.195181 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.notifications [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81305) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 11:19:39.196673 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1479/2938] 10.222.0.46 () {58 vars in 1233 bytes} [Fri Oct 28 11:19:39 2022] DELETE /identity/v3/projects/58e7c57cf57a44a990735d6b86d4e736 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 11:19:46.893456 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Authenticating user token {{(pid=81306) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:19:46.894996 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:46.983490 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6bd47-d422-40c1-94d3-8ece96923047 admin admin] Validating token access rules against request {{(pid=81306) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:19:46.986945 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:47.044417 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81306) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:19:47.045022 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] REQUEST_METHOD: `GET` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:19:47.045300 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] SCRIPT_NAME: `/identity` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:19:47.045573 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81306) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:19:47.047327 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.fernet_utils [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81306) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:19:47.058274 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:19:47.059844 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abde67-da3c-4876-b457-c25bd6a3f79c service placement] RBAC: Authorization granted {{(pid=81306) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:19:47.069531 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: [pid: 81306|app: 0|req: 1460/2939] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:19:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:19:49.191285 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=10,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:19:57.065413 dv-cinder-zed-3-iscsi devstack@keystone.service[81306]: DEBUG dbcounter [-] [81306] Writing DB stats keystone:SELECT=29 {{(pid=81306) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 11:24:50.893032 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Authenticating user token {{(pid=81305) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 11:24:50.894501 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.fernet_utils [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81305) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 11:24:50.909693 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49032513-154b-431b-b629-3a21e4adf350 admin admin] Validating token access rules against request {{(pid=81305) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 11:24:50.914589 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '73aefca6b8d04a0e8bd2fad4ffedbccd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f6834f875cac433ca5419f5b85701542', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81305) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 11:24:50.915189 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] REQUEST_METHOD: `GET` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 11:24:50.916952 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] SCRIPT_NAME: `/identity` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 11:24:50.917244 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81305) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 11:24:50.926767 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 11:24:50.928392 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8efdc7-72a7-43fa-a59c-74f29557f97b service placement] RBAC: Authorization granted {{(pid=81305) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 11:24:50.938561 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: [pid: 81305|app: 0|req: 1480/2940] 10.222.0.46 () {60 vars in 1365 bytes} [Fri Oct 28 11:24:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 11:25:00.934186 dv-cinder-zed-3-iscsi devstack@keystone.service[81305]: DEBUG dbcounter [-] [81305] Writing DB stats keystone:SELECT=6 {{(pid=81305) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}